Du2016 - Overview of Deep Learning
Du2016 - Overview of Deep Learning
:XKDQ&KLQD1RYHPEHU
2YHUYLHZRI'HHS/HDUQLQJ
Abstract²,Q UHFHQW \HDUV GHHS OHDUQLQJ KDV DFKLHYHG JUHDW SHUIRUPDQFH RI WUDGLWLRQDO PDFKLQH OHDUQLQJ PHWKRGV XVXDOO\
VXFFHVV LQ PDQ\ ILHOGV VXFK DV FRPSXWHU YLVLRQ DQG QDWXUDO UHO\RQXVHUV¶H[SHULHQFHVZKLOH GHHSOHDUQLQJDSSURDFKHVUHO\
ODQJXDJH SURFHVVLQJ &RPSDUHG WR WUDGLWLRQDO PDFKLQH OHDUQLQJ RQ WKH GDWD 7KHUHIRUH ZH FDQ ILQG RXW WKDW GHHS OHDUQLQJ
PHWKRGVGHHSOHDUQLQJKDVDVWURQJOHDUQLQJDELOLW\DQGFDQPDNH DSSURDFKHV KDYH UHGXFHG WKH GHPDQGV IRU XVHUV :LWK WKH
EHWWHU XVH RI GDWDVHWV IRU IHDWXUH H[WUDFWLRQ %HFDXVH RI LWV SURJUHVV RI FRPSXWHU WHFKQRORJ\ FRPSXWHUV¶ SHUIRUPDQFH LV
SUDFWLFDELOLW\GHHSOHDUQLQJEHFRPHVPRUHDQGPRUHSRSXODUIRU UDSLGO\LPSURYHG0HDQZKLOHLQIRUPDWLRQRQWKH,QWHUQHWLVDOVR
PDQ\UHVHDUFKHUVWRGRUHVHDUFKZRUNV,QWKLVSDSHUZHPDLQO\ VSHZLQJ RXW 7KHVH IDFWRUV SURYLGH D VWURQJ LPSHWXV IRU GHHS
LQWURGXFH VRPH DGYDQFHG QHXUDO QHWZRUNV RI GHHS OHDUQLQJ DQG
OHDUQLQJ WR GHYHORS DQG PDNH GHHS OHDUQLQJ EHFRPH WKH
WKHLU DSSOLFDWLRQV %HVLGHV ZH DOVR GLVFXVV WKH OLPLWDWLRQV DQG
SUHYDOHQWPHWKRGLQPDFKLQHOHDUQLQJ
SURVSHFWVRIGHHSOHDUQLQJ
,Q WKLV SDSHU ZH PDNH D V\VWHPDWLF LQWURGXFWLRQ IRU GHHS
Keywords—deep learning; machine learning; neural network OHDUQLQJ IURP PDQ\ DVSHFWV E\ H[SDWLDWLQJ LWV UHVHDUFK
SURJUHVVHV VWDWHRIWKHDUW PRGHOV IUDPHZRUNV DQG
, ,1752'8&7,21 DSSOLFDWLRQV UHVSHFWLYHO\ )LUVW ZH LQWURGXFH WKH UHVHDUFK
'HHSOHDUQLQJZDVGHYHORSHGIURPDUWLILFLDOQHXUDOQHWZRUN SURJUHVVHVLQ6HFWLRQ,,7KHQZHLQWURGXFHVHYHUDOW\SLFDOGHHS
DQGQRZLWLVDSUHYDOHQWILHOGRIPDFKLQHOHDUQLQJ7KHUHVHDUFK OHDUQLQJ PRGHOV LQ 6HFWLRQ ,,, DQG VHYHUDO GHHS OHDUQLQJ
RIDUWLILFLDOQHXUDOQHWZRUNEHJDQIURPV0F&XOORFK et al IUDPHZRUNV LQ 6HFWLRQ ,9 1H[W ZH OLVW VRPH DSSOLFDWLRQV RI
>@SURSRVHGWKH0F&XOORFK3LWWV 03 PRGHOE\DQDO\]LQJDQG GHHS OHDUQLQJ LQ 6HFWLRQ 9 )LQDOO\ ZH FRQFOXGH WKLV SDSHU LQ
VXPPDUL]LQJ WKH FKDUDFWHULVWLFV RI QHXURQV +HEE et al >@ 6HFWLRQ9,
SURSRVHG D FHOO DVVHPEO\ WKHRU\ WR H[SODLQ WKH DGDSWDWLRQ RI
FHUHEUDOQHXURQGXULQJWKHOHDUQLQJSURFHVV7KLVWKHRU\KDGDQ ,, 5(6($5&+352*5(66(6
LPSRUWDQW LQIOXHQFH RQ WKH GHYHORSPHQW RI QHXUDO QHWZRUNV 7KHFRQFHSWRIGHHSOHDUQLQJZDVSXWIRUZDUGLQDWILUVW
7KHQ 5RVHQEODWW et al >@ LQYHQWHG WKH SHUFHSWURQ DOJRULWKP $IWHUWKDWGHHSOHDUQLQJLVVWLOOFRQWLQXDOO\GHYHORSLQJDWDEURDG
7KLV DOJRULWKP LV D NLQG RI ELQDU\ FODVVLILHU ZKLFK EHORQJV WR $WSUHVHQWWKHUHDUHPDQ\RXWVWDQGLQJILJXUHVVXFKDV*HRIIUH\
VXSHUYLVHG OHDUQLQJ :LGURZ SURSRVHG WKH DGDSWLYH OLQHDU +LQWRQ<RVKXD%HQJLR<DQQ/H&XQDQG$QGUHZ1J7KH\DUH
HOHPHQWDQGLWLVDVLQJOHOD\HUDUWLILFLDOQHXUDOQHWZRUNEDVHG OHDGLQJWKHUHVHDUFKGLUHFWLRQRIGHHSOHDUQLQJ6RPHFRPSDQLHV
RQWKH03PRGHO8QIRUWXQDWHO\0LQVN\DQG3DSHUWSRLQWHGWKDW OLNH *RRJOH DQG )DFHERRN KDYH PDGH ORWV RI UHVHDUFK
WKH SHUFHSWURQ DOJRULWKP KDG JUHDW OLPLWDWLRQV LQ WKHRU\ DQG DFKLHYHPHQWVLQGHHSOHDUQLQJDQGDSSOLHGWKHPWRYDULRXVILHOGV
PDGHDQHJDWLYHHYDOXDWLRQRQWKHSURVSHFWVRIQHXUDOQHWZRUNV ,QWKLV\HDU*RRJOH¶V$OSKD*RSURJUDPGHIHDWHG/HH6HGROLQ
ZKLFK OHG WKH GHYHORSPHQW RI QHXUDO QHWZRUNV WR KLW D QDGLU *RFRPSHWLWLRQZKLFKVKRZHGWKDWGHHSOHDUQLQJKDGDVWURQJ
+RZHYHU+RSILHOGet al>@SURSRVHGWKH+RSILHOGQHWZRUNLQ OHDUQLQJ DELOLW\ :KDW¶V PRUH *RRJOH¶V 'HHS'UHDP LV DQ
WKH HDUO\ V 7KLV PDGH DUWLILFLDO QHXUDO QHWZRUN UHYLYHG H[FHOOHQW VRIWZDUH ZKLFK FDQ QRW RQO\ FODVVLI\ LPDJHV EXW
7KHQ+LQWRQet al. >@SURSRVHGWKH%ROW]PDQQPDFKLQHE\XVLQJ JHQHUDWH VWUDQJH DQG DUWLILFLDO SDLQWLQJV EDVHG RQ LWV RZQ
VLPXODWHG DQQHDOLQJ DOJRULWKP ,Q WKH V YDULRXV VKDOORZ NQRZOHGJH )DFHERRN DQQRXQFHG D QHZ DUWLILFLDO LQWHOOLJHQFH
PDFKLQHOHDUQLQJPHWKRGVZHUHSURSRVHGRQHDIWHUDQRWKHUVXFK V\VWHPQDPHG'HHS7H[W'HHS7H[WLVDGHHSOHDUQLQJEDVHG
DV VXSSRUW YHFWRU PDFKLQH >@ %RRVWLQJ >@ 'XH WR WKH WH[WXQGHUVWDQGLQJHQJLQHZKLFKFDQFODVVLI\PDVVLYHDPRXQWV
DGYDQWDJHVRIWKHVHPHWKRGVERWKLQWKHRU\DQGLQDSSOLFDWLRQ RIGDWDSURYLGHFRUUHVSRQGLQJVHUYLFHVDIWHULGHQWLI\LQJXVHUV¶
DUWLILFLDOQHXUDOQHWZRUNKLWDQDGLUDJDLQ$IWHU+LQWRQHWDOSXW FKDWWLQJPHVVDJHVDQGFOHDQXSVSDPPHVVDJHV
IRUZDUGWKHFRQFHSWRIGHHSOHDUQLQJLQWKHMRXUQDO6FLHQFHLQ
DUWLILFLDOQHXUDOQHWZRUNRQFHDJDLQUHFHLYHGPXFKLQWHUHVW 'HHS OHDUQLQJ VWDUWHG UHODWLYHO\ ODWH EXW GHYHORSHG YHU\
IURPWKHUHVHDUFKFRPPXQLW\ UDSLGO\ DW KRPH 7KHUH KDYH DFKLHYHG UHPDUNDEOH SURJUHVV LQ
FROOHJHVXQLYHUVLWLHVUHVHDUFKLQVWLWXWHVDQGFRPSDQLHV%DLGX
'HHSOHDUQLQJPRGHOVXVXDOO\DGRSWKLHUDUFKLFDOVWUXFWXUHV KDV HVWDEOLVKHG D GHHS OHDUQLQJ LQVWLWXWH WR H[SORUH KRZ WR
WR FRQQHFW WKHLU OD\HUV 7KH RXWSXW RI D ORZHU OD\HU FDQ EH FRPSOHWH PDQ\ D WDVN ZLWK GHHS OHDUQLQJ %DLGX¶V XQPDQQHG
UHJDUGHG DV WKH LQSXW RI D KLJKHU OD\HU YLD VLPSOH OLQHDU RU JURXQG YHKLFOH KDV DFFRPSOLVKHG URDG WHVW XQGHU FRPSOLFDWHG
QRQOLQHDU FDOFXODWLRQV 7KHVH PRGHOV FDQ WUDQVIRUP ORZOHYHO URDG FRQGLWLRQV ,)/<7(. VWDUWHG WKH UHVHDUFK RI VSHHFK
IHDWXUHVRIWKHGDWDLQWR KLJKOHYHODEVWUDFWIHDWXUHV2ZQLQJ WR UHFRJQLWLRQ EDVHG RQ 'HHS 1HXUDO 1HWZRUN '11 LQ
WKLV FKDUDFWHULVWLF GHHS OHDUQLQJ PRGHOV FDQ EH VWURQJHU WKDQ 7KH\ ODXQFKHG WKH ILUVW RQOLQH &KLQHVH VSHHFK UHFRJQLWLRQ
VKDOORZPDFKLQHOHDUQLQJPRGHOVLQIHDWXUHUHSUHVHQWDWLRQ7KH V\VWHP DQG DQ DGYDQFHG WHFKQRORJ\ WR UHFRJQL]H GLIIHUHQW
7KLVZRUNLVVXSSRUWHGLQSDUWE\WKH1DWLRQDO1DWXUDO6FLHQFH )RXQGDWLRQ
RI&KLQDXQGHU*UDQWV
,(((
ODQJXDJHV$QGQRZWKH\KDYHSXEOLVKHGDKLJKSHUIRUPDQFH
\ /[[ÿ
FRPSXWLQJ+3&SODWIRUPLQFRRSHUDWLRQZLWK,QWHO
,,, '((3/($51,1*02'(/6
)URP WKH EHJLQQLQJ WR WKH SUHVHQW WKHUH DUH D ORW RI GHHS
OHDUQLQJPRGHOV 7KHW\SLFDOPRGHOVLQFOXGH$XWRHQFRGHU$(
'HHS %HOLHI 1HWZRUN '%1 &RQYROXWLRQDO 1HXUDO 1HWZRUN
&11DQG5HFXUUHQW1HXUDO1HWZRUN511,QWKLVVHFWLRQ
ZHPDLQO\LQWURGXFHVRPHVWDWHRIWKHDUWPRGHOV [ [ÿ
%RRVWLQJ
)RUHVWVZKLFKLVDQRYHOVWUXFWXUHWKDWXQLILHVFODVVLILFDWLRQWUHHV
ZLWK&11,QWKHLUSDSHUWKHQHWZRUNVWUXFWXUHLVYHU\FOHDUWKDW
6HOHFWHG3DWFKHV
WKH\ UHSODFH WKH VRIWPD[ OD\HU ZLWK D VWRFKDVWLF DQG
GLIIHUHQWLDEOHGHFLVLRQWUHHPRGHO7KHGHFLVLRQWUHHLVDNLQGRI
WUHHVWUXFWXUHG FODVVLILHU ZKLFK FRQVLVWV RI GHFLVLRQ QRGHV DQG
SUHGLFWLRQQRGHV7KH GHFLVLRQQRGHVGHFLGHWKHURXWHVWKDWKRZ
VDPSOHVSDVV DORQJWKHWUHH7KHSUHGLFWLRQQRGHVDUHWRFDOFXODWH
)HHGIRUZDUG
)HHGIRUZDUG
%DFNIRUZDUG
)HHGIRUZDUG
V ZDV FRQVLGHUHG DV WKH SUHGHFHVVRU RI &11 7KH DIWHUWKHWZRLPDJHVDUHSURFHVVHGE\WKHILUVWILYHOD\HUV7KHQ
UHPDUNDEOHFKDUDFWHULVWLFRI&11LVWKDWWKHQHWZRUNXVHVWKH WKH\ DUH LQSXWWHG WR WKH IROORZLQJ VHW RI OD\HU VR WKDW WKH
ORFDO UHFHSWLYH ILHOG DQG ZHLJKW VKDULQJ %\ XVLQJ WKHVH WZR SUREDELOLW\RIDVXFFHVVIXOJUDVS
VWUDWHJLHV WKH QXPEHU RI WUDLQLQJ SDUDPHWHUV LV UHGXFHG D. Recurrent Neural Network
VLJQLILFDQWO\ ZKLFK FDQ PDNH WKH QHWZRUN EHFRPHV OHVV
FRPSOLFDWHG $ W\SLFDO &11 VWUXFWXUH FRQVLVWV RI VRPH 5HFXUUHQW 1HXUDO 1HWZRUN LV D NLQG RI DUWLILFLDO QHXUDO
FRQYROXWLRQDOOD\HUVSRROLQJOD\HUVDQGIXOO\FRQQHFWHGOD\HUV QHWZRUN $SDUW IURP KDYLQJ WKH VWUXFWXUH RI WKH IHHGIRUZDUG
7KHFRQYROXWLRQDOOD\HULVXVHGIRUIHDWXUHH[WUDFWLRQ(DFKLQSXW QHXUDO QHWZRUN WKHUH H[LVWV GLUHFWHG F\FOHV LQ 511 7KLV
RIWKHQHXURQLQWKLVOD\HULVFRQQHFWHGWRDORFDOUHFHSWLYHILHOG VWUXFWXUHDOORZVWKHLQIRUPDWLRQWREHFLUFXODWHGLQWKHQHWZRUN
RI WKH SUHYLRXV RQH 7KH SRROLQJ OD\HU LV XVHG IRU IHDWXUH VR WKH RXWSXW RI HDFK WLPH LV QRW RQO\ UHODWHG WR WKH LQSXW DW
PDSSLQJ ,W FDQ UHGXFH WKH GLPHQVLRQ RI GDWD DQG EH DEOH WR SUHVHQWEXWUHODWHGWRWKHLQSXWDW SUHYLRXVWLPHVWDPSV
PDLQWDLQWKHLQYDULDQFHRIWKHQHWZRUNVWUXFWXUH $OWKRXJKWKHWUDGLWLRQDO511LVDEOHWRGHDOZLWKWLPHVHULHV
,QUHFHQW\HDUV&11KDVJRWORWVRIDWWHQWLRQVIURPPDQ\ GDWDWKHUHH[LVWVDVHULRXVSUREOHPDERXWJUDGLHQWYDQLVKLQJLQ
UHVHDUFKHUV,WLVDQH[FHOOHQWPRGHOWKDWFDQDFFRPSOLVKWDVNV WKH SURFHVV RI EDFN SURSDJDWLRQ 7KHUHIRUH 511 FDQ RQO\ EH
HIILFLHQWO\ 7KHUH DUH PDQ\ W\SHV RI &11 VWUXFWXUHV VXFK DV XVHGIRUVKRUWWHUPPHPRU\LQPRVWFDVHV,QRUGHUWRVROYHWKLV
/H1HW >@ $OH[1HW >@ =)1HW >@ 9**1HW >@ DQG SUREOHPPDQ\UHVHDUFKHUVEHJDQWRSXWIRUZDUGVHYHUDONLQGV
*RRJOH1HW>@/H&XQHWDOSURSRVHGDFRQYROXWLRQDOQHXUDO RI LPSURYHG VWUXFWXUHV VXFK DV /RQJ 6KRUW7HUP 0HPRU\
QHWZRUNQDPHO\/H1HWDQGDSSOLHGWRKDQGZULWLQJUHFRJQLWLRQ /670 'LIIHUHQW IURP WKH WUDGLWLRQDO 511 /670 KDV D
$OH[1HWLVPDLQO\XVHGWRREMHFWGHWHFWLRQV$IWHUWKDW=)1HW PHPRU\FHOODQGDQLQSXWRXWSXWJDWHVWUXFWXUH7KHPHPRU\FHOO
9**1HWDQG*RRJOH1HWZHUHSXWIRUZDUGEDVHGRQ$OH[1HW LV XVHG WR UHFRUG LQIRUPDWLRQ DQG WKH LQSXWRXWSXW JDWH
$WSUHVHQW&11LVVWLOODQDFWLYHWRSLF ZLWKPDQ\GLUHFWLRQVWR GHWHUPLQHVZKHWKHUWKHLQIRUPDWLRQLVFDSDEOHRIIORZLQJLQWRRU
H[SORUH 6RPH UHVHDUFKHUV ZDQW WR LQFUHDVH WKH FRPSOH[LW\ RI RXWRIWKHPHPRU\FHOO'XHWRWKHVHFKDUDFWHULVWLFV/670KDV
&11 VWUXFWXUHV 2WKHUV ZDQW WR FRPELQH &11 ZLWK RWKHU DEHWWHUSHUIRUPDQFHWKDQ511LQORQJWHUPPHPRU\WDVNV
WUDGLWLRQDOPDFKLQHOHDUQLQJV %\HRQ et al >@ SURSRVHG D FRPSOHWHO\ OHDUQLQJEDVHG
/LXet al>@SURSRVHGDQ66'PRGHOZKLFKFRXOGGHWHFW DSSURDFKIRUVFHQHODEHOLQJXVLQJDNLQGRI'/670UHFXUUHQW
REMHFWVHIILFLHQWO\ZLWKKLJKDFFXUDF\7KHPRGHOFRQVLVWVRID QHXUDOQHWZRUNV7KHQHWZRUNLVGLYLGHGLQWRWKUHHPDLQOD\HUV
WUXQFDWHG EDVH QHWZRUN VWUXFWXUH DQG DX[LOLDU\ VWUXFWXUH 7KH LQSXW OD\HU KLGGHQ OD\HU DQG RXWSXW OD\HU 7KH LQSXW LPDJH LV
WUXQFDWHGEDVHQHWZRUNLQ>@DGRSWV9**DQGWKHDX[LOLDU\ VSOLWLQWRVHYHUDOQRQRYHUODSSLQJZLQGRZVWRWKHLQSXWQHWZRUN
VWUXFWXUHDGRSWVVRPHIHDWXUHOD\HUVWRWKHHQGRI9**7KH 7KHKLGGHQOD\HUFRQVLVWVRID'/670OD\HUDQGIHHGIRUZDUG
QHWZRUN FDQ SURGXFH D VHW RI IL[HGVL]H ERXQGLQJ ER[HV IURP OD\HU 7KH ' /670 OD\HU LV XVHG WR PHPRUL]H FRQWH[W
PDQ\IHDWXUHPDSV,WFDQDOVRJLYHFDWHJRU\VFRUHVLIWKHUHLVDQ LQIRUPDWLRQ LQ DOO GLUHFWLRQV DQG WKH IHHGEDFN OD\HU FRPELQHV
REMHFWLQWKHERXQGLQJER[HVDQGFRUUHVSRQGLQJRIIVHWV:KHQ LQIRUPDWLRQ WRJHWKHU 7KH RXWSXW OD\HU QRUPDOL]HV WKH RXWSXWV
WUDLQLQJ66'WKHORVVIXQFWLRQZKLFKLVWKHZHLJKWHGVXPRIWKH IURPWKHODVWKLGGHQOD\HUZLWKDVRIWPD[IXQFWLRQDQGJHQHUDWHV
ORFDOL]DWLRQ ORVV DQG FRQILGHQFH ORVV ZLOO EH SURGXFHG RQ SUREDELOLWLHV DERXW ZKLFK FODVVHV WKH WDUJHWV EHORQJ WR
IRUZDUG SURSDJDWLRQ /DVWO\ WKH ORVV IXQFWLRQ FDQ EH XVHG WR ([SHULPHQWDO UHVXOWV VKRZ WKH HIIHFWLYHQHVV RI WKH SURSRVHG
ILQHWXQHWKHPRGHORQEDFNSURSDJDWLRQ7KHVWUXFWXUHRIWKLV PRGHO7KHQHWZRUNLVVKRZQLQ)LJ
PRGHOLVVKRZQLQ)LJ /LX et al >@ LQWURGXFHG D QHZ DSSURDFK WR MRLQWO\ OHDUQ
.RQWVFKLHGHU et al >@ SURSRVHG 'HHS 1HXUDO 'HFLVLRQ IHDWXUHUHSUHVHQWDWLRQVDFURVVPXOWLSOHUHODWHGWDVNV7KHQRYHOW\
([WUD)HDWXUH/D\HUV
9**
WKURXJK3RROOD\HU
1RQ0D[LPXP6XSSUHVVLRQ
&ODVVLILHU&RQY[[[FODVVHV
'HWHFWLRQVSHU&ODVV
&ODVVLILHU&RQY[[[FODVVHV
66'
G G
G G G G
)LJ 7KHWUHHPRGHORI'HHS1HXUDO'HFLVLRQ)RUHVWVFRQVLVWVRIVHYHUDOGHFLVLRQWUHHV7KHGHFLVLRQQRGHVGHFLGHKRZVDPSOHVSDVVWKURXJKWKHWUHHV7KH
SUHGLFWLRQQRGHV SURGXFHWKHGHFLVLRQSUREDELOLWLHVDIWHUVDPSOHVUHDFKWKHFRUUHVSRQGLQJILQDO QRGHV$GDSWHGIURP>@
wi Q [Q[Q [Q[Q
Q
/670 /670 /670 /670
/670 /670
Ȉ
/670 /670
Ȉ ı 2XWSXW
6RIWPD[ 3UF _ Z i
7 0
[Q[Q [Q[Q
6
)HHGIRUZDUGOD\HU
,QSXW,N '/670OD\HU
+LGGHQOD\HU
xP xP xP xTP
H[WUDFWLRQ PHWKRG QHHGV XVHUV WR GHVLJQ ZKDW IHDWXUHV WKH\
y m
VKRXOGH[WUDFW$QGWKHVHSURFHVVHVDUHRIWHQKLJKFRVWDQGWLPH
VRIWPD[
xV xV xV xTV
FRQVXPLQJ'HHSOHDUQLQJKDVWKHDELOLW\RIXQVXSHUYLVHGIHDWXUH
h s h s h s hT s OHDUQLQJDQGLWFDQH[WUDFWWKHIHDWXUHVRILPDJHVZLWKRXWDQ\
KXPDQ LQWHUYHQWLRQ 7KXV LW LV JUDGXDOO\ DWWUDFWHG PRUH DQG
xQ xQ xQ xTQ PRUHDWWHQWLRQE\UHVHDUFKHUV$IWHU.UL]KHYVN\et al>@JRWD
xV xV xV xTV
VRIWPD[ yn EUHDNWKURXJK E\XVLQJ&11LQ,PDJH1HW /695&GHHS
D0RGHO, OHDUQLQJEHFRPHVPRUHDQGPRUHSRSXODULQFRPSXWHUYLVLRQDQG
KDYH PDGH DQ H[FHOOHQW EUHDNWKURXJK XS WR QRZ 6R IDU WKH
x x x xT
PHWKRGZKLFKFRQMXQFWVWKUHHUHVLGXDO,QFHSWLRQQHWZRUNV ZLWK
RQH,QFHSWLRQY >@PDNHVWKHLPDJHUHFRJQLWLRQWDVNDFKLHYH
hP hP hP hT s VRIWPD[ y m WRSHUURU /HDUQHG0LOOHUet al>@SURSRVHGDGHHS
OHDUQLQJ PHWKRG ZKLFKPDGHWKH DFFXUDF\RI IDFHUHFRJQLWLRQ
ULVH WR DERXW $W SUHVHQW WKH UHVHDUFKHUV LQ WKH &KLQHVH
hQ hQ hQ hTQ VRIWPD[ yn 8QLYHUVLW\ RI +RQJ .RQJ KDYH LQFUHDVHG WKH IDFH UHFRJQLWLRQ
DFFXUDF\DERYH>@
x x x xT 'HHSOHDUQLQJKDVEHHQLQFRQWLQXRXVGHYHORSPHQWLQQDWXUDO
E0RGHO,, ODQJXDJH SURFHVVLQJ DQG JRW PDQ\ DFKLHYHPHQWV LQ PDQ\
DSSOLFDWLRQVLQFOXGLQJVSHHFKUHFRJQLWLRQ VSHHFKV\QWKHVLVDQG
x x x xT 4XHVWLRQ$QVZHULQJ 7KH WUDGLWLRQDO VSHHFK UHFRJQLWLRQ
V\VWHPV ZHUH PRVWO\ EDVHG RQ *DXVVLDQ 0L[WXUH 0RGHO DQG
hP hP hP hT s VRIWPD[ y m +LGGHQ 0DUNRY 0RGHO LQ WKH SDVW IRU D ORQJ WLPH +RZHYHU
WKHVHPHWKRGV FRXOGQRWGHDOZLWKGHHSFKDUDFWHULVWLFVZHOODQG
hQ hQ hQ hTQ DUHVHQVLWLYHWRGLVWXUEDQFHVIURPWKHRXWVLGHHQYLURQPHQW$IWHU
DGRSWLQJGHHSOHDUQLQJLQVSHHFKUHFRJQLWLRQWKHSHUIRUPDQFHV
hQ hQ hQ hTQ VRIWPD[ yn RI WKH V\VWHPV KDYH LPSURYHG GUDPDWLFDOO\ 1RZ WKH VSHHFK
UHFRJQLWLRQV\VWHP'HHS6SHHFKZKLFKLVGHVLJQHGE\%DLGX
x x x xT KDV UHGXFHG WKH HUURU UDWH WR LQ &KLQHVH VSHHFK WHVW $W
SUHVHQW *RRJOH 'HHS0LQG SXEOLVKHG D QHZ VSHHFK V\QWKHVLV
F0RGHO,,,
V\VWHPZKLFKZDVQDPHG:DYH1HW >@:DYH1HWLVDNLQGRI
)LJ 7KUHHDUFKLWHFWXUHVRIWKHQHWZRUNIRUWH[WFODVVLILFDWLRQ ZLWKPXOWL GHHS QHXUDO QHWZRUN DQG FDQ JHQHUDWH UDZ DXGLR ZDYHIRUPV
WDVNOHDUQLQJ $GDSWHGIURP>@ &RPSDUHG WR RWKHU WH[WWRVSHHFK V\VWHPV :DYH1HW FDQ
7RUFK >@ FDQ VXSSRUW PRVW RI WKH PDFKLQH OHDUQLQJ JHQHUDWHPRUHUHDOLVWLFVRXQGVDVZHOODVPXVLF)URP'HHS0LQG
DOJRULWKPV,WLQFOXGHVPRVWSRSXODU DOJRULWKPVDQGPRGHOV VXFK LW VKRZHG WKDW :DYH1HW UHGXFHG WKH JDS EHWZHHQ KXPDQ DQG
DV PXOWLOD\HU SHUFHSWURQV VXSSRUW YHFWRU PDFKLQHV *DXVVLDQ V\QWKHVL]HG YRLFHV E\ RYHU LQ (QJOLVK DQG &KQLHVH
PL[WXUH PRGHOV KLGGHQ 0DUNRY PRGHOV VSDWLDO DQG WHPSRUDO 4XHVWLRQ$QVZHULQJ 4$ LVDKRWUHVHDUFKGLUHFWLRQRIQDWXUDO
FRQYROXWLRQDOQHXUDOQHWZRUNV $GD%RRVW %D\HVFODVVLILHUV DQG ODQJXDJH SURFHVVLQJ ZKLFK FDQ JLYH D FRUUHFW DQG FRQFLVH
VR RQ %HVLGHV VXSSRUWLQJ &38 DQG *38 7RUFK DOVR FDQ EH DQVZHU ZLWK WKH QDWXUDO ODQJXDJH IRUP IRU QDWXUDO ODQJXDJH
HPEHGGHGLQWRL26$QGURLGDQG)3*$ SUREOHPV 7KHYLFWRU\RI :DWVRQ>@ RQMHRSDUG\KDVVKRZQ
WKDW4$EDVHGRQGHHSOHDUQLQJKDVLWVRZQXQLTXHVXSHULRULW\
7KHDQR >@ LVDIUDPHZRUNEDVHGRQ3\WKRQ,W FDQVXSSRUW
VRPHXQVXSHUYLVHGDQGVHPLVXSHUYLVHGOHDUQLQJDSSURDFKHVDV 9, &21&/86,21
ZHOO DV VXSHUYLVHG OHDUQLQJ DSSURDFKHV VXFK DV ORJLVWLF
'HHSOHDUQLQJDSSURDFKHVDUHSUDFWLFDOIRUXVWRVROYHPDQ\
UHJUHVVLRQ PXOWLOD\HUSHUFHSWURQGHHS&11$(5%0DQG
SUREOHPV,QWKLVSDSHUZHLQWURGXFHGHHSOHDUQLQJPRGHOVDQG
'%17KDQNVWRWKHVHIXQFWLRQV7KHDQRLVXVXDOO\EHXVHGIRU
IUDPHZRUNVLQGHWDLO'HHSOHDUQLQJGLIIHUHQWNLQGVRIPRGHOV
WHDFKLQJ DW DERDUG +RZHYHU 7KHDQR KDV D ZHDNQHVV WKDW LWV
DQG IUDPHZRUNV DQG LW KDV KDG PDQ\ DSSOLFDWLRQV LQ PDQ\
VSHHGLVWRR VORZ
DVSHFWV)URPWKHVHZHFDQVHHWKDWGHHSOHDUQLQJKDVDJUHDW
9 $33/,&$7,2162) '((3/($51,1* GHYHORSPHQWSRWHQWLDO
$IWHUZHGLVFXVVWKHVHPRGHOVDQGIUDPHZRUNVZHFDQILQG ,QIXWXUHLWLVIRUHVHHDEOHWKDWGHHSOHDUQLQJFRXOGHVWDEOLVK
WKDW GHHS OHDUQLQJ DSSURDFKHV FRXOG KHOS XV WR DFKLHYH SHUIHFW WKHRULHV WR H[SODLQ LWV SHUIRUPDQFHV 0HDQZKLOH LWV
SHUIRUPDQFHV LQ YDULRXV DSSOLFDWLRQV ,Q WKLV VHFWLRQ ZH DELOLWLHVRIXQVXSHUYLVHGOHDUQLQJZLOOEHHQKDQFHGVLQFHWKHUH
LQWURGXFHVRPHDSSOLFDWLRQVRIGHHSOHDUQLQJLQFRPSXWHUYLVLRQ DUHPLOOLRQVRI GDWDLQWKHZRUOGEXWLWLVQRWDSSOLFDEOHWRDGG
DQGQDWXUDOODQJXDJHSURFHVVLQJ ODEHOV WR DOO RI WKHP ,W LV DOVR SUHGLFWHG WKDW QHXUDO QHWZRUN
VWUXFWXUHV ZLOO EHFRPH PRUH FRPSOH[ VR WKDW WKH\ FDQ H[WUDFW
'HHS OHDUQLQJ KDV KDG D ZLOG GHYHORSPHQW LQ FRPSXWHU PRUH VHPDQWLFDOO\ PHDQLQJIXO IHDWXUHV :KDW¶V PRUH GHHS
YLVLRQ VXFK DV REMHFW GHWHFWLRQ REMHFW WUDFNLQJ DQG LPDJH OHDUQLQJZLOOFRPELQHZLWKUHLQIRUFHPHQWOHDUQLQJEHWWHUDQGZH
VHJPHQWDWLRQ 2EMHFW GHWHFWLRQ DLPV WR UHFRJQL]H D FODVV RI FDQXVHWKLVDGYDQWDJHVWRDFFRPSOLVKPRUHWDVNV
REMHFWV IURP D ODUJH QXPEHU RI LPDJHV 7KH WUDGLWLRQDO REMHFW
GHWHFWLRQ PHWKRGV PDLQO\ LQFOXGH FDQGLGDWH UHJLRQ VHOHFWLRQ
IHDWXUH H[WUDFWLRQ DQG FODVVLILFDWLRQ 7KLV PDQXDO IHDWXUH
5()(5(1&(6
>@ :60F&XOORFKDQG:3LWWV³$ORJLFDOFDOFXOXVRIWKHLGHDVLPPDQHQW >@ .6LPRQ\DQDQG$=LVVHUPDQ³9HU\GHHSFRQYROXWLRQDOQHWZRUNVIRU
LQQHUYRXVDFWLYLW\´The bulletin of mathematical biophysicsYROSS ODUJHVFDOHLPDJHUHFRJQLWLRQ´ arXiv preprint arXiv
>@ & 6]HJHG\ : /LX < -LD DQG 3 6HUPDQHW ³*RLQJ GHHSHU ZLWK
>@ '2+HEE³7KHRUJDQL]DWLRQRIEHKDYLRU´J. Appl. Behav. Anal.YRO FRQYROXWLRQV´Proceedings of the IEEE Conference on Computer Vision
SS± and Pattern Recognition BostonSS
>@ ) 5RVHQEODWW ³7KH SHUFHSWURQ D SUREDELOLVWLF PRGHO IRU LQIRUPDWLRQ >@ :/LX'$QJXHORY'(UKDQ&6]HJHG\DQG65HHG³66'6LQJOH
VWRUDJHDQGRUJDQL]DWLRQLQWKHEUDLQ´Psychol. Rev.YROQR SS 6KRW0XOWL%R['HWHFWRU´arXiv preprint arXiv: 1512.02325
>@ 3.RQWVFKLHGHU0)LWHUDX$&ULPLQLVLDQG65%XOR³'HHS1HXUDO
'HFLVLRQ)RUHVWV´IEEE International Conference on Computer Vision
>@ - - +RSILHOG ³1HXUDO QHWZRUNV DQG SK\VLFDO V\VWHPV ZLWK HPHUJHQW
SantiagoSS
FROOHFWLYHFRPSXWDWLRQDODELOLWLHV´P. Natl. Acad. Sci. USAYROSS
>@ 6/HYLQH33DVWRU$.UL]KHYVN\DQG'4XLOOHQ³/HDUQLQJ+DQG(\H
>@ '+$FNOH\*(+LQWRQDQG7-6HMQRZVNL³$OHDUQLQJDOJRULWKPIRU &RRUGLQDWLRQIRU5RERWLF*UDVSLQJZLWK'HHS/HDUQLQJDQG/DUJH6FDOH
EROW]PDQQPDFKLQHV´Cognitive Sci.YROSS 'DWD&ROOHFWLRQ´arXiv preprint arXiv: 1603.02199
>@ &&RUWHVDQG99DSQLN³6XSSRUW9HFWRU1HWZRUNV´Mach. Learn.YRO >@ :%\HRQ70%UHXHO)5DXHDQG0/LZLFNL³6FHQHODEHOLQJZLWK
SS /670UHFXUUHQWQHXUDOQHWZRUNV´Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition Boston SS
>@ <)UHXQGDQG5(6FKDSLUH³$GHFLVLRQWKHRUHWLFJHQHUDOL]DWLRQRIRQ
OLQHOHDUQLQJDQGDQDSSOLFDWLRQWRERRVWLQJ´J. comput. Syst. Sci.YRO
>@ 3 /LX ; 4LX DQG ; +XDQJ ³5HFXUUHQW 1HXUDO 1HWZRUN IRU 7H[W
QRSS
&ODVVLILFDWLRQ ZLWK 0XOWL7DVN /HDUQLQJ´ arXiv preprint arXiv:
>@ 39LQFHQW+/DURFKHOOH<%HQJLRDQG3$0DQ]DJRO³([WUDFWLQJ 1605.05101
DQG FRPSRVLQJ UREXVW IHDWXUHV ZLWK GHQRLVLQJ DXWRHQFRGHUV´
Proceedings of the 25th international conference on Machine learning >@ < -LD( 6KHOKDPHU- 'RQDKXH6 .DUD\HY- /RQJ5 *LUVKLFNet al
New York pp. 1096-1103 ³&DIIH &RQYROXWLRQDO DUFKLWHFWXUH IRU IDVW IHDWXUH HPEHGGLQJ´
Proceedings of the 22nd ACM international conference on Multimedia
>@ $ 1J ³6SDUVH DXWRHQFRGHU´ CS294A Lecture Notes Stanford Univ. OrlandoSS
CaliforniaSS
>@ 0 $EDGL $$JDUZDO3 %DUKDP ( %UHYGR = )&KHQ&&LWURet al
>@ < ;LRQJ DQG 5 =XR ³5HFRJQLWLRQ RI JHRFKHPLFDO DQRPDOLHV XVLQJ D ³7HQVRUIORZ/DUJHVFDOHPDFKLQHOHDUQLQJRQKHWHURJHQHRXVGLVWULEXWHG
GHHS DXWRHQFRGHU QHWZRUN´ Comput. Geosci.-UK YRO SS V\VWHPV´arXiv preprint arXiv: 1603.04467
>@ 5 &ROOREHUW 6 %HQJLR DQG - 0DULpWKR] ³7RUFK D PRGXODU PDFKLQH
>@ 3/LX6+DQ=0HQJDQG<7RQJ³)DFLDO([SUHVVLRQ5HFRJQLWLRQYLD OHDUQLQJVRIWZDUHOLEUDU\´ Idiap No. EPFL-REPORT-82802
D%RRVWHG'HHS%HOLHI1HWZRUN´Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition Columbus SS >@ 5$O5IRX* $ODLQ $$OPDKDLUL & $QJHUPXHOOHU'%DKGDQDX1
%DOODV et al ³7KHDQR $ 3\WKRQ IUDPHZRUN IRU IDVW FRPSXWDWLRQ RI
PDWKHPDWLFDOH[SUHVVLRQV´arXiv preprint arXiv: 1605.02688
>@ 6 .LP % 3DUN %6 6RQJ DQG 6 <DQJ ³'HHS EHOLHI QHWZRUN EDVHG
VWDWLVWLFDO IHDWXUH OHDUQLQJ IRU ILQJHUSULQW OLYHQHVV GHWHFWLRQ´ Pattern >@ & 6]HJHG\ 6 ,RIIH 9 9DQKRXFNH DQG $ $OHPL ³,QFHSWLRQY
Recogn. Lett YRO SS LQFHSWLRQUHVQHW DQG WKH LPSDFW RI UHVLGXDO FRQQHFWLRQV RQ OHDUQLQJ´
arXiv preprint arXiv: 1602.07261
>@ .)XNXVKLPD60L\DNHDQG7,WR³1HRFRJQLWURQ$QHXUDOQHWZRUN
>@ * % +XDQJ + /HH DQG ( /HDUQHG0LOOHU ³/HDUQLQJ KLHUDUFKLFDO
PRGHOIRUDPHFKDQLVPRIYLVXDOSDWWHUQUHFRJQLWLRQ´IEEE Trans. Syst.
UHSUHVHQWDWLRQV IRU IDFH YHULILFDWLRQ ZLWK FRQYROXWLRQDO GHHS EHOLHI
Man Cybern.YROSS
QHWZRUNV´Proceedings of the IEEE Conference on Computer Vision and
>@ </HFXQ/%RWWRX<%HQJLRDQG3+DIIQHU³*UDGLHQWEDVHGOHDUQLQJ Pattern Recognition, 5KRGH,VODQG, pp. 2
DSSOLHG WR GRFXPHQW UHFRJQLWLRQ´Proc. IEEE YRO SS
>@ <6XQ;:DQJDQG;7DQJ³'HHSO\OHDUQHGIDFHUHSUHVHQWDWLRQVDUH
VSDUVH VHOHFWLYH DQG UREXVW´ Proceedings of the IEEE Conference on
>@ $ .UL]KHYVN\ 6XWVNHYHU , 6XWVNHYHU DQG *( +LQWRQ ³,PDJHQHW Computer Vision and Pattern Recognition BostonSS
FODVVLILFDWLRQ ZLWK GHHS FRQYROXWLRQDO QHXUDO QHWZRUNV´ Proc. Neural
>@ $9' 2RUG6 'LHOHPDQ + =HQ . 6LPRQ\DQ 2 9LQ\DOV $*UDYHV
Information and Processing SystemsSS
et al³:DYH1HW $*HQHUDWLYH0RGHOIRU5DZ$XGLR´arXiv preprint
>@ 0'=HLOHUDQG5)HUJXV³9LVXDOL]LQJDQGXQGHUVWDQGLQJFRQYROXWLRQDO arXiv: 1609.03499
QHWZRUNV´European Conference on Computer Vision ZurichSS
>@ ' )HUUXFFL $ /HYDV 6 %DJFKL ' *RQGHNDQG(7 0XHOOHU ³:DWVRQ
%H\RQG-HRSDUG\´Artif. Intell. YROSS