1

Video Classification and Detection

CVPR 2019 Tutorial

Christoph Feichtenhofer
Facebook AI Research (FAIR)
2

Task: Human action classification & detection

C. Feichtenhofer, H. Fan, J. Malik, K. He SlowFast Networks for Video Recognition. Tech. Report, arXiv, 2018
3

Johansson: Perception of Biological Motion

Sources: Johansson, G. “Visual perception of biological motion and a model for its analysis.” Perception & Psychophysics. 14(2):201-211. 1973.
4

Motivation: Separate visual pathways in nature

è Dorsal stream (‘where’) recognizes motion and locates
objects
OPTICAL FLOW STIMULI

è “Interconnection”
e.g. in STS area

è Ventral (‘what’) stream performs object recognition

Sources: “Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli." Journal of neurophysiology 65.6 (1991).
“A cortical representation of the local visual environment”, Nature. 392 (6676): 598–601, 2009
https://en.wikipedia.org/wiki/Two-streams_hypothesis
5

Background: Two-Stream Convolutional Networks

Individual processing of spatial and temporal information

• Using a separate 2D (x,y) ConvNet recognition stream for each
• Late fusion via softmax score averaging

K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014
 6

Background: Two-Stream Network Fusion and 2D -> 3D Transformation/Inflation

Motion Stream

conv2_x

conv3_x

conv4_x

conv5_x
conv1

loss
+

+
x x x x
Appearance Stream

conv2_x

conv3_x

conv4_x

conv5_x
conv1

loss
+

+
o ST-ResNet allows the hierarchical learning of spacetime features by connecting the
appearance and motion channels of a two-stream architecture.

C. Feichtenhofer, A. Pinz, and A. Zisserman. Convolutional two-stream network fusion for video action recognition. In Proc. CVPR, 2016
C. Feichtenhofer, A. Pinz, and R. Wildes. Spatiotemporal residual networks for video action recognition. In NIPS, 2016.
C. Feichtenhofer, A. Pinz, and R. Wildes. Spatiotemporal multiplier networks for video action recognition. In CVPR, 2017.
7

Background: Transforming spatial networks into temporal ones by Inflation

Time t
t
… …

conv1 conv1 conv1 conv1 conv1 conv1 conv1 conv1 conv1

* * * * * * * * * *
res2 res2 res2 res2 res2 res2 res2 res2 res2

+ + + + + + + + +
* * * * * * * * * *
res3 res3 res3 res3 res3 res3 res3 res3 res3

+ + + + + + + + +
* * * * * * * * * *
res4 res4 res4 res4 res4 res4 res4 res4 res4

+ + + + + + + + +
* * * * * * * * * *
res5 res5 res5 res5 res5 res5 res5 res5 res5

+ + + + + + + + +

pool

fc

o Inflation allows to transform spatial filters to spatiotemporal ones (3D or 2D spatial +1D temporal)
C. Feichtenhofer, A. Pinz, and R. Wildes. Spatiotemporal residual networks for video action recognition. In NIPS, 2016.
J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. CVPR, 2017.
D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
8

Background: Transforming 2D networks into 3D by Inflation

C. Feichtenhofer, A. Pinz, and R. Wildes. Spatiotemporal residual networks for video action recognition. In NIPS, 2016.
J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. CVPR, 2017.
 9

Visualizing the learned representation

o Regularized activation maximization on the input

Input Convolutional Feature Maps of VGG-16

Spatial regularization Temporal
weight regularization weight

Motion

Loss
height
width

Fusion
*
depth

c
Maximize channel c
Appearance

e.g. c = 4
C. Feichtenhofer, A. Pinz, and R. Wildes, A. Zisserman. What have we learned from deep representations for action recognition?. In CVPR, 2018.
C. Feichtenhofer, A. Pinz, and R. Wildes, A. Zisserman. Deep insights into convolutional networks for video recognition?. In IJCV, 2019.
 10

Visualizing the learned representation

o Slow
Regularized
motion activation maximization on the input
Fast motion
(high temporal reg.) (low temporal reg.) Maximum Activation

Punch

Input Convolutional Feature Maps of VGG-16

Drumming
Spatial regularization Temporal
weight
StillRings regularization weight
BoxingSpeedBag

Motion JumpingJack

IceDancing

BenchPress Loss
height
width
Archery

Fusion
*
depth
MilitaryParade

BoxingPunchingBag
c
0 5 10 15
Maximize channel
20 25 30 35
c
Appearance

e.g. c = 4
C. Feichtenhofer, A. Pinz, and R. Wildes, A. Zisserman. What have we learned from deep representations for action recognition?. In CVPR, 2018.
C. Feichtenhofer, A. Pinz, and R. Wildes, A. Zisserman. Deep insights into convolutional networks for video recognition?. In IJCV, 2019.
12

Filter #251 at conv5 fusion – the strongest

local Billiards unit
slow medium fast

Maximum Activation

Billiards

TableTennisShot

SoccerJuggling

Fencing

FieldHockeyPenalty

Basketball

Lunges

BaseballPitch

SoccerPenalty

FloorGymnastics

0 5 10 15 20 25 30 35 40
13

Last layer
è “Billiards”

Appearance Slow motion Fast motion

e.g. “ball rolling” e.g. “player
moving”
14

Last layer
è
“CleanAndJerk”
Appearance Slow motion Fast motion
e.g. “shaking with e.g. “push bar”
bar”

Fast motion appearance

15

Background: 3D Convolutional Networks

Input clip of ~2sec H,W
T

prediction
“Head-butting”
(Kinetics classificaiton annotation)

G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler. Convolutional learning of spatio-temporal features. In Proc. ECCV, 2010.
D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3D convolutional networks. In Proc. ICCV, 2015.
J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. CVPR, 2017.
16

Background: Non-Local Convolutional Network Blocks

o Self-attention in the spatiotemporal domain allows long-range feature aggregation

X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proc. CVPR, 2018.
17

Background: Limited temporal input length of 3D ConvNets

Actions

3D CNN

2-4 seconds
 19

New work: Long-Term Feature Banks for Video Understanding

Actions

FBO
Long-Term Feature Bank
... ...
3D CNN

short clip

full video

... ... ... ...

CY Wu, C. Feichtenhofer, H. Fan, K. He, P. Krähenbühl, R. Girshick Long-Term Feature Banks for Detailed Video Understanding. In Proc. CVPR, 2019.
20

This talk: SlowFast Networks for Video Recognition

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik and Kaiming He
• New backbone network for human action classification & detection

Slow pathway

H,W
T

Fast pathway

C. Feichtenhofer, H. Fan, J. Malik, K. He SlowFast Networks for Video Recognition. Tech. Report, arXiv, 2018
 21

Motivation: Separate visual pathways for what and where

Magno cell properties1 Parvo cell properties1

è Minority of cells in è Majority of cells in

LGN: ~20% LGN: ~80%
è Large receptive field è Small receptive field
è High contrast è Able to differentiate
sensitivity detailed stimuli
è Able to differentiate è Color Sensitive
only coarse stimuli è Processes
è Color Blind information about
è Processes color & detail
information about è Slow conduction rate
depth & motion (less myelin)
è Fast conduction rate
(more myelin)
David C. Van Essen, Jack L. Gallant, Neural mechanisms of form and
1 https://www.ucalgary.ca/pip369/mod2/visualpathways/magnoparvo
motion processing in the primate visual system,
Neuron, Volume 13, Issue 1, July 1994, Pages 1-10, ISSN 0896-6273
22

The basic idea of SlowFast

• The network consists of two pathways:
• (i) a Slow pathway, operating at low frame rate, to capture spatial semantics
• (ii) a Fast pathway, operating at high frame rate, to capture motion at fine
temporal resolution
23

The basic idea of SlowFast networks

Slow pathway

Slow
C T
C T
C T

prediction
H,W
T

Fast
αT
αT βC
βC
αT
βC Fast pathway
“Hand-clap” e.g. α = 8
(AVA detection annotation) β = 1/8
26

Example instantiation of a SlowFast network

• Dimensions are
• Strides are {temporal, spatial2}
• The backbone is ResNet-50
• Residual blocks are shown by brackets
• Non-degenerate temporal filters are
underlined
• Here the speed ratio is α = 8 and the
channel ratio is β = 1/8
• Orange numbers mark fewer channels,
for the Fast pathway
• Green numbers mark higher temporal
resolution of the Fast pathway
• No temporal pooling is performed
throughout the hierarchy
27

SlowFast training recipe, Kinetics action classification

• Kinetics has 240k training videos and 20k validation videos in 400 classes
• Our training recipe for training without ImageNet initialization (inflation)
• T = input size, τ = temporal stride
 28

SlowFast ablations: Individual paths

• Kinetics action classification dataset has 240k training
Slow pathway
videos and 20k validation videos in 400 classes

prediction
C T
C T
C T
H,W
T

prediction
α=8
β = 1/8 αT
αT
βC
βC
αT
βC Fast pathway
 29

SlowFast ablations: Lateral fusion, Kinetics action classification

• Kinetics dataset has 240k training videos and 20k
validation videos in 400 classes Slow pathway

C T
C T
C T

prediction
H,W
T

𝑡 C

αT
αT βC
βC
αT
βC Fast pathway
30

SlowFast ablations: Learning curves

 31

SlowFast ablations: Making the Fast path thin in channel dimension

• Kinetics dataset has 240k training videos and 20k Slow pathway
validation videos in 400 classes

C T
C T
C T

prediction
H,W
T

αT
αT βC
βC
αT
βC Fast pathway
33
Conv1 filters

SlowFast ablatios: Weak input

Slow

β = 1/4
rgb grayscale time diff

β = 1/6 β = 1/8

Fast
β = 1/16
β = 1/32
t t t

t t t t

rgb grayscale dt
35

SlowFast ablations: Temporal sampling rates

Slow pathway

C T
C T
C T

prediction
H,W
T

αT
αT βC
βC
αT
βC Fast pathway
37

SlowFast: State-of-the-art comparison on Kinetics

+ 5.1% at 10%
§§§§§§§§§

top-1 of FLOPs

37
38

SlowFast: State-of-the-art comparison Kinetics-600

• Kinetics-600 has 392k training videos and 30k validation videos in 600 classes
39

SlowFast: State-of-the-art comparison Charades1

• Charades has 9.8k training videos and 1.8k validation videos in 157 classes
• Multi-label classification setting of longer activities spanning 30 seconds on average

1G.A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing data collection for activity
understanding. In ECCV, 2016. , CVPR 2016
 40

Experiments: AVA1 Action Detection SlowFast detector output

• Fine-scale localization of 80 different

physical actions
• Data from 437 different movies and
spatiotemporal labels are provided in a
1Hz interval
• 211k training and 57k validation video
segments
• We follow the standard protocol of
evaluating on 60 most freqent classes
• Every person is annotated with a
bounding box and (possibly multiple)
actions

1Gu et al. AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions, CVPR 2018
41

Experiments: AVA Action Detection

Slow pathway

Slow
T RPN1
T C
C
C T
H,W

concat
T Detections

C RoIAlign

Fast
αT
αT βC
βC
αT
βC Fast pathway

1Faster R-CNN with a ResNeXt-101-FPN backbone pretrained on COCO keypoints

42

SlowFast
SlowFast:ablations:
State-of-the-art
AVA action
comparison
detection
AVA

+5.2mAP

Slow pathway

T RPN
T C
C
C T
H,W +6.4mAP +13mAP

concat
T Det
34.25mAP
C RoIAlign

αT
αT βC
βC
αT
βC Fast pathway
46

SlowFast ablations: AVA class level performance

47

Experiments: AVA Qualitative results

48

Experiments: AVA Qualitative results

49

Conclusion
Slow pathway
• The time axis is a special dimension of video.
• 3D ConvNets treat space and time uniformly.
• SlowFast and Two-Stream networks share motivation
from biological studies.
• We investigate an architecture design that focuses on
contrasting the speed along the temporal axis.
• The SlowFast architecture achieves state-of-the-art
accuracy for video action classification and detection
without need of any (e.g. ImageNet) pretraining.
• Given the mutual benefits of jointly modeling video
Fast pathway
with different temporal speeds, we hope that this
concept can foster further research in video analysis.
FAIR Research Engineer
Menlo Park, CA
Seattle, WA

ACCELERATE AND SCALE CV RESEARCH

Familiarity with CV and ML
Ability to write high-quality and performance-critical code

[email protected]