med_poster_spie
0 likes•120 views
The document presents a system for detecting complex events in unconstrained videos using pre-trained deep CNN models. Frame-level features extracted from various CNNs are fused to form video-level descriptors, which are then classified using SVMs. Evaluation on a large video corpus found that fusing different CNNs outperformed individual CNNs, and no single CNN worked best for all events as some are more object-driven while others are more scene-based. The best performance was achieved by learning event-dependent weights for different CNNs.
![Acknowledgements: The authors wish to thank Edward Scott, Kevin Brady, and Charlie K. Dagli for their technical contributions.
Pre-trained Deep CNN Models for Detecting Complex Events in
Unconstrained VideosRapid event detection faces an emergent need to process large videos collections;
whether surveillance or unconstrained web videos, automatic recognition of high-level,
complex events is challenging. In response to the complicated, computationally
demanding, non-replicable pre-existing methods, we designed a simple system that is
quick, effective and carries minimal overhead in terms of memory and storage. Most
importantly, our system is clearly described, modular in nature, and easily
reproducible. We explored various pre-trained Convolutional Neural Networks (CNN)
as off-the-shelf feature extractors, as both stand-alone and fused with others. For this,
we used a large corpus of unconstrained, real-world video data. Frame-level features
were fused to form video-level descriptors by means of both early and late fusion.
Several insights were found on using pre-trained CNNs as off-the-shelf feature
extractors for the task of event detection. Fusing SVMs with different CNNs revealed
some interesting facts, finding some combinations to be complimentary. It was found
that no single CNN works best for all events, as some events are more object- driven
while others are more scene-based. Our top performance resulted from learning
event-dependent weights for different CNNs.
Abstract
[1] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. Caffe:
Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014).
[2] Vedaldi, A., & Fulkerson, B. (200 8). VLFeat: An Open and Portable Library of Computer Vision Algorithms. In
ACM MM (2010).
[3] Wang ,L., Guo ,S., Huang ,W., and Qiao,Y., Places205-vgg net models for scene recognition. CoRR
abs/1508.01667 (2015).
[4] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L., Imagenet: A large-scale hierarchical image
database. In Proceedings of IEEE CVPR (2009).
[5] Zhou,B., Lapedriza,A., Xiao,J., Torralba,A., & Oliva,A. Learning deep features for scene recognition using places
database. NIPS (2014).
[6] Simonyan, K. and Zisserman, A., Very deep convolutional networks for large-scale image recognition. CoRR
abs/1409.1556 (2014).
[7] Robinson, J. P., Scott, E., and Fu, Y., NEU MIT-LL @ TRECVid 2015: Multimedia Event Detection by Deep
Feature Learning. In Proceedings of TRECVID 2015 (2015).
[8] Over, P., Awad, G., Michel, M., Fiscus, J., Kraaij, W., Smeaton, A. F., Quenot, G., and Ordelman, R., Trecvid 2015
– an overview of the goals, tasks, data, evaluation mechanisms and metrics. In Proceedings of TRECVID (2015).
[9] Hsu, C.W. & Lin, C.J. A comparison of methods for multiclass support vector machines. IEEE Transactions on
Neural Networks (2002).
[10] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A.,
Bernstein, M., Berg, A. C., & Fei-Fei, L., Imagenet large scale visual recognition challenge. IJCV (April 2015).
[11] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A.,
Going deeper with convolutions. In Proceedings of IEEE CVPR (2015).
References
CNN Frame Features Video Feature One-vs-rest SVMs
(a) (b) (c) (d) (e)
Image
conv64
conv64
maxpool
conv128
conv128
maxpool
conv256
conv256
maxpool
conv512
conv512
maxpool
FC-4096
FC-4096
FC-1000
softmax
conv512
conv512
maxpool
…
Average Pooling 𝒳"
Video Frames
Fig 1 A single system fusing across modalities: multi-modal to exploit subtle connections which enables deeper interfaces.
Fig 3 List of events included in MED15 (a). Each comes with an event kit: e.g., event 21 (b), consisting of a text
description (top) and video exemplars (bottom). Notice the ambiguity, i.e., its intra-class variations being broad in scope.
Evidential description explicitly lists different Scenes, Objects, and Activities. Note that listed entities may or may not exist.
E021
Event name: Attempting a bike trick
Definition: One or more people attempt to do a trick on a bicycle, motorcycle, or other type of motorized bike. To count as a
bike for purposes of this event, the vehicle must have two wheels (excludes unicycles, ATVs, etc.).
Explication: Bikes are normally ridden with a person sitting down on seat and holding onto the handlebars and steering with
their hands. Tricks consist of difficult ways of riding the bike, such as on one wheel, steering with feet or standing on the seat;
or intentional motions made with the bike that are not simply slowing down/stopping the bike, propelling it forward, or steering
the bike as it moves. Steering around obstacles or steering a bike off of a jump and landing on the ground are generally not
considered tricks in and of themselves, however if the bike jump is set up so that the person is jumping over something, (e.g.
jumping over people or vehicles or over a river), or if the person does a flip or other trick in the air, that would be considered a
trick.
Evidential description:
Scene: outside, often in a skate park, parking lot or street
Objects/ People: person riding a bike, bike, ramps, helmet, concrete floor, audiences
Activities: riding bike on one wheel, standing on top of bike, jumping with the bike (especially over or onto objects),
spinning or flipping bike
Audio: sounds of bike hitting surface during the trick, audience cheering
Text Descriptions
Exemplar Videos
• All experiments were conducted using pipeline shown in Fig 4.
• Table 1 lists descriptions of each run and MAP scores, revealing the following:
o VGG-16 CNN’s fc8-layer (Run 2) tops that of the fc7 (Run 1).
o Modeling the object distributions discriminates (Run2) better than scene
classes (Run3).
o Object and Scene fused together early (Run 6) yields best performance.
o Hybrid-CNN (Run 4) does not compare well to two separate CNN models.
• Table 2 displays class labels of maximum SVM weights for 4 extreme cases, 2
where objects did relatively well and 2 not well.
Results
Table 1 List of runs with descriptions and resulting MAP scores.
• Multimedia Analysis data typically demands experts of each modality [see Fig 1].
• Internet archives continues to see an increasing pool of accessible videos that
demand better technologies for automatic tagging, search & retrieval [see Fig 2].
Motivation
• YouTube has more than 1 billion users
• Every day people watch hundreds of millions of
hours on YouTube and generate billions of views
• 300 hours of video are uploaded to
YouTube every minute
NEEDS
An efficient way to sort, search and retrieve this huge
amount of video based on CONTENT
Fig 2 YouTube report for year 2015.
• MED task is supported by TRECVid in the form of an annual workshop, where data
and metrics needed for a laboratory style evaluation are provided [see Fig3].
Fig 4 The work flow of our MED system. Given an input video, it is first sampled at approximately one fps (a). Sample frames are then fed forward through pre-trained CNN models (b),
yielding a set of frame-level feature vectors (c). Next, a video-level descriptor is formed by average pooling the frame-level features (d). Video descriptor is projected to a non-linear space
via RBF kernel, and then passed to the one-vs-rest SVM models (e).
Joseph P. Robinson and Yun Fu
(a) (b)
All-Terrain Bike
PonchoHoneycomb
E021
Unicycle Tricycle
Bee House
Theater Curtain Home Theater
All-Terrain Bike
Volleyball
Disc Brake Tandem Bike
E031E028E033
Top Activation 2nd Activation 3rd Activation
Vehiclewith1 wheel
drivenby pedals
Vehiclewith3 wheels
drivenby pedals
Bike w sturdy frame + fat
tires; designedfor
mountainouscountry
Shed containinga
numberof beehives
Tiny hex-cellsofbeeswax
usedby bees to store
honey+ larvae
Blanket-likecloakwitha
holecenteredfor head
Cloth, stagefront; opens
to start + closesto
break/end
TV + videoequipmentto
providetheatermovie
experienceathome
An inflatedballusedin
playingvolleyball
Bike w sturdy frame + fat
tires; designedfor
mountainouscountry
Brakes that applies
friction to spinningdisk
via brakepads
Bike with2 sets of pedals
and2 seats
Table 2 Class labels of max SVM weights. E021 and E031 scored the highest using VGG-16 CNN (object) features (top 2
rows); VGG-16 did not perform well on E028 and E033 (bottom 2 rows).
(a)
(b)
(c)
(d)
Fig 5 Frames nearest SVM weights:TP (green boxes)
and FP (red boxes), E21(a) E31(b) E28(c) E33 (d).
Fig 6 Notional capability. Example interface for facilitate multimodal search and retrieval capabilities.
• Incorporate multimodality capability for cross media searches [see Fig 6].
• Integrate temporal information into models.
• Train set of object classes that is event specific.
Conclusions/Future Work
Descriptions MAP(%)
Run 1 fc7 features, VGG model (4,096D) 12.4
Run 2 fc8 features, VGG model (1,000D) 13.3
Run 3 fc8 features, Places205 model (205D) 8.4
Run 4 fc8 features, Hybrid-CNN model (1,186D) 11.7
Run 5 fc8 features, VGG (1,000D) + Places205 (205D) models; average SVM scores 11.8
Run 6 fc8 features, VGG + Places205 models concatenated (1,250D) 16.0
This material is based upon work supported by the U.S. Department of Homeland Security, Science and Technology Directorate, Officeof University Programs, under Grant Award 2013-ST-061-ED0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security. [10/2013]](https://image.slidesharecdn.com/9a25ca64-50ee-45ed-8b5f-d5e778e77886-160418173413/85/med_poster_spie-1-320.jpg)
med_poster_spie
- 1. Acknowledgements: The authors wish to thank Edward Scott, Kevin Brady, and Charlie K. Dagli for their technical contributions. Pre-trained Deep CNN Models for Detecting Complex Events in Unconstrained VideosRapid event detection faces an emergent need to process large videos collections; whether surveillance or unconstrained web videos, automatic recognition of high-level, complex events is challenging. In response to the complicated, computationally demanding, non-replicable pre-existing methods, we designed a simple system that is quick, effective and carries minimal overhead in terms of memory and storage. Most importantly, our system is clearly described, modular in nature, and easily reproducible. We explored various pre-trained Convolutional Neural Networks (CNN) as off-the-shelf feature extractors, as both stand-alone and fused with others. For this, we used a large corpus of unconstrained, real-world video data. Frame-level features were fused to form video-level descriptors by means of both early and late fusion. Several insights were found on using pre-trained CNNs as off-the-shelf feature extractors for the task of event detection. Fusing SVMs with different CNNs revealed some interesting facts, finding some combinations to be complimentary. It was found that no single CNN works best for all events, as some events are more object- driven while others are more scene-based. Our top performance resulted from learning event-dependent weights for different CNNs. Abstract [1] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014). [2] Vedaldi, A., & Fulkerson, B. (200 8). VLFeat: An Open and Portable Library of Computer Vision Algorithms. In ACM MM (2010). [3] Wang ,L., Guo ,S., Huang ,W., and Qiao,Y., Places205-vgg net models for scene recognition. CoRR abs/1508.01667 (2015). [4] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L., Imagenet: A large-scale hierarchical image database. In Proceedings of IEEE CVPR (2009). [5] Zhou,B., Lapedriza,A., Xiao,J., Torralba,A., & Oliva,A. Learning deep features for scene recognition using places database. NIPS (2014). [6] Simonyan, K. and Zisserman, A., Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014). [7] Robinson, J. P., Scott, E., and Fu, Y., NEU MIT-LL @ TRECVid 2015: Multimedia Event Detection by Deep Feature Learning. In Proceedings of TRECVID 2015 (2015). [8] Over, P., Awad, G., Michel, M., Fiscus, J., Kraaij, W., Smeaton, A. F., Quenot, G., and Ordelman, R., Trecvid 2015 – an overview of the goals, tasks, data, evaluation mechanisms and metrics. In Proceedings of TRECVID (2015). [9] Hsu, C.W. & Lin, C.J. A comparison of methods for multiclass support vector machines. IEEE Transactions on Neural Networks (2002). [10] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., & Fei-Fei, L., Imagenet large scale visual recognition challenge. IJCV (April 2015). [11] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A., Going deeper with convolutions. In Proceedings of IEEE CVPR (2015). References CNN Frame Features Video Feature One-vs-rest SVMs (a) (b) (c) (d) (e) Image conv64 conv64 maxpool conv128 conv128 maxpool conv256 conv256 maxpool conv512 conv512 maxpool FC-4096 FC-4096 FC-1000 softmax conv512 conv512 maxpool … Average Pooling 𝒳" Video Frames Fig 1 A single system fusing across modalities: multi-modal to exploit subtle connections which enables deeper interfaces. Fig 3 List of events included in MED15 (a). Each comes with an event kit: e.g., event 21 (b), consisting of a text description (top) and video exemplars (bottom). Notice the ambiguity, i.e., its intra-class variations being broad in scope. Evidential description explicitly lists different Scenes, Objects, and Activities. Note that listed entities may or may not exist. E021 Event name: Attempting a bike trick Definition: One or more people attempt to do a trick on a bicycle, motorcycle, or other type of motorized bike. To count as a bike for purposes of this event, the vehicle must have two wheels (excludes unicycles, ATVs, etc.). Explication: Bikes are normally ridden with a person sitting down on seat and holding onto the handlebars and steering with their hands. Tricks consist of difficult ways of riding the bike, such as on one wheel, steering with feet or standing on the seat; or intentional motions made with the bike that are not simply slowing down/stopping the bike, propelling it forward, or steering the bike as it moves. Steering around obstacles or steering a bike off of a jump and landing on the ground are generally not considered tricks in and of themselves, however if the bike jump is set up so that the person is jumping over something, (e.g. jumping over people or vehicles or over a river), or if the person does a flip or other trick in the air, that would be considered a trick. Evidential description: Scene: outside, often in a skate park, parking lot or street Objects/ People: person riding a bike, bike, ramps, helmet, concrete floor, audiences Activities: riding bike on one wheel, standing on top of bike, jumping with the bike (especially over or onto objects), spinning or flipping bike Audio: sounds of bike hitting surface during the trick, audience cheering Text Descriptions Exemplar Videos • All experiments were conducted using pipeline shown in Fig 4. • Table 1 lists descriptions of each run and MAP scores, revealing the following: o VGG-16 CNN’s fc8-layer (Run 2) tops that of the fc7 (Run 1). o Modeling the object distributions discriminates (Run2) better than scene classes (Run3). o Object and Scene fused together early (Run 6) yields best performance. o Hybrid-CNN (Run 4) does not compare well to two separate CNN models. • Table 2 displays class labels of maximum SVM weights for 4 extreme cases, 2 where objects did relatively well and 2 not well. Results Table 1 List of runs with descriptions and resulting MAP scores. • Multimedia Analysis data typically demands experts of each modality [see Fig 1]. • Internet archives continues to see an increasing pool of accessible videos that demand better technologies for automatic tagging, search & retrieval [see Fig 2]. Motivation • YouTube has more than 1 billion users • Every day people watch hundreds of millions of hours on YouTube and generate billions of views • 300 hours of video are uploaded to YouTube every minute NEEDS An efficient way to sort, search and retrieve this huge amount of video based on CONTENT Fig 2 YouTube report for year 2015. • MED task is supported by TRECVid in the form of an annual workshop, where data and metrics needed for a laboratory style evaluation are provided [see Fig3]. Fig 4 The work flow of our MED system. Given an input video, it is first sampled at approximately one fps (a). Sample frames are then fed forward through pre-trained CNN models (b), yielding a set of frame-level feature vectors (c). Next, a video-level descriptor is formed by average pooling the frame-level features (d). Video descriptor is projected to a non-linear space via RBF kernel, and then passed to the one-vs-rest SVM models (e). Joseph P. Robinson and Yun Fu (a) (b) All-Terrain Bike PonchoHoneycomb E021 Unicycle Tricycle Bee House Theater Curtain Home Theater All-Terrain Bike Volleyball Disc Brake Tandem Bike E031E028E033 Top Activation 2nd Activation 3rd Activation Vehiclewith1 wheel drivenby pedals Vehiclewith3 wheels drivenby pedals Bike w sturdy frame + fat tires; designedfor mountainouscountry Shed containinga numberof beehives Tiny hex-cellsofbeeswax usedby bees to store honey+ larvae Blanket-likecloakwitha holecenteredfor head Cloth, stagefront; opens to start + closesto break/end TV + videoequipmentto providetheatermovie experienceathome An inflatedballusedin playingvolleyball Bike w sturdy frame + fat tires; designedfor mountainouscountry Brakes that applies friction to spinningdisk via brakepads Bike with2 sets of pedals and2 seats Table 2 Class labels of max SVM weights. E021 and E031 scored the highest using VGG-16 CNN (object) features (top 2 rows); VGG-16 did not perform well on E028 and E033 (bottom 2 rows). (a) (b) (c) (d) Fig 5 Frames nearest SVM weights:TP (green boxes) and FP (red boxes), E21(a) E31(b) E28(c) E33 (d). Fig 6 Notional capability. Example interface for facilitate multimodal search and retrieval capabilities. • Incorporate multimodality capability for cross media searches [see Fig 6]. • Integrate temporal information into models. • Train set of object classes that is event specific. Conclusions/Future Work Descriptions MAP(%) Run 1 fc7 features, VGG model (4,096D) 12.4 Run 2 fc8 features, VGG model (1,000D) 13.3 Run 3 fc8 features, Places205 model (205D) 8.4 Run 4 fc8 features, Hybrid-CNN model (1,186D) 11.7 Run 5 fc8 features, VGG (1,000D) + Places205 (205D) models; average SVM scores 11.8 Run 6 fc8 features, VGG + Places205 models concatenated (1,250D) 16.0 This material is based upon work supported by the U.S. Department of Homeland Security, Science and Technology Directorate, Officeof University Programs, under Grant Award 2013-ST-061-ED0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security. [10/2013]