Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
5 likes16,404 views
The document discusses a methodology for temporal activity detection in untrimmed videos using recurrent neural networks and convolutional neural networks. It covers the architecture, training processes, and results of classifying and detecting various activities across a large dataset of nearly 20,000 videos. The authors also highlight challenges, future work in end-to-end training, and attention models for improved performance.
Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
- 1. Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks Alberto Montes July 15th, 2016 Xavi Giró Amaia Salvador
- 2. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 2
- 3. Motivation 3
- 4. Motivation 4
- 7. Problem Definition 7 Videos Activity Temporal Localization Longboarding
- 11. 11 Large-Scale Activity Recognition Challenge Stats: ● 19,994 Videos ● 200 Activities ● 660 hours of video ● 313 hours of activities ● 65.6 million of frames
- 12. Dataset 12
- 13. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 13
- 18. 3D Convolution 18 Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015, December). Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE ICCV 2015 (pp. 4489-4497). IEEE.
- 19. 3D Convolution 19 ● 16-frame video clip as input ● 80 million parameters ● 3x3x3 filter size at all conv layers Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015, December). Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE ICCV 2015 (pp. 4489-4497). IEEE.
- 22. Segments Proposals 22 Shou, Z., Wang, D., & Chang, S. F. Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs CVPR 2016.
- 24. RNN for Activity Localization 24 Yeung, Serena, Olga Russakovsky, Greg Mori, and Li Fei-Fei. et al. "End-to-end Learning of Action Detection from Frame Glimpses in Videos." CVPR 2016
- 25. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 25
- 26. Architecture Overview 26 16 frames 200 activities + background 16 frames 200 activities + background 16 frames 200 activities + background
- 27. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 27
- 28. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 28
- 29. C3D Network 29 Caffe + by feature vector published on:
- 31. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 31
- 32. Audio Features 32 C3D Recurrent Neural Network Input Audio Features: ● MFCC ● Spectral concatvideo features Provided by Ignasi Esquerra
- 33. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 33
- 36. Network Architecture 36 LSTM with previous output feedback
- 37. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 37
- 38. Training Methodology Categorical Cross Entropy Loss 38
- 39. Training Methodology For unbalanced data, weighted loss: 39 660 hours of video 313 hours of activities
- 40. Outline 3. Methodology a. Extracting C3D Features b. Audio Features c. Network Architecture d. Training Methodology e. Post-Processing 40
- 41. Classification Post-Processing 41 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN
- 42. Classification Post-Processing 42 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN Average
- 43. Classification Post-Processing 43 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN Average Max Probability
- 44. Detection Post-Processing 44 Background Activity 1 Activity 2 Activity 200 Clip1 Clip2 Clip3 ClipN Applied a mean filter of k samplestime
- 47. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 47
- 48. Classification: Audio Features 48 mAP = 0.5755mAP = 0.5938 Music unrelated to the activity is often added to the videos in post-processing, causing a decrease in performance when audio and video features are combined.
- 49. Classification: Depth Analysis 49 mAP = 0.5938 mAP = 0.5492 mAP = 0.5635 Deeper networks present overfitting
- 50. Classification Results Per Activity 50
- 51. Classification Results Per Activity 51 Using the Pommel Horse Sailing Playing Ice Hockey Rock Climbing BMX
- 52. Classification Results Per Activity 52 Drinking Coffee Peeling Potatoes Having an Ice Cream Rock-Paper-Scissors Polishing shoes
- 54. Detection 54 mAP = 0.2251 mAP = 0.2067 Model with feedback did not improve results
- 55. Training with feedback 55 512-LSTM video features0 0 1 0 0 0 concat When training previous ground truth
- 56. Training with feedback 56 512-LSTM video features0 0.1 0.6 0.2 0.1 0 concat When testing previous prediction
- 57. Comparing Post-Processing 57 Ɣ Grid search for optimal parameters
- 58. Detection Results per Activity 58
- 59. Detection Results per Activity 59 Windsurfing Riding Bumper Cars Playing Racquetball Using the Pommel Horse Using Parallel Bars
- 60. Detection Results per Activity 60 Drinking Coffee Putting on Shoes Rock-Paper-Scissors Removing Curlers Smoking a Cigarette
- 62. Qualitative Evaluation 62 Ground Truth: Playing water polo Prediction: 0.765 Playing water polo 0.202 Swimming 0.007 Springboard diving
- 63. Qualitative Evaluation 63 Ground Truth: Hopscotch Prediction: 0.848 Running a marathon 0.023 Triple jump 0.022 Javelin throw
- 66. Challenge Results 66 Classification Task (24 participants) Baseline 42.20% 0% 100% 93.23% Winner Average Performance 66.26%58.74% UPC Team * results over test subset Slide Design by Issey Masuda mAP
- 67. Challenge Results 67 Detection Task (6 participants) Baseline 9.70% 0% 50% 42.47% Winner Average Performance 29.94%22.36% UPC Team mAP * results over test subset Slide Design by Issey Masuda
- 68. Outline 1. Introduction 2. Related Work 3. Methodology 4. Results 5. Conclusions and Future Work 68
- 70. Conclusions 70 Video Spatial Net Temporal Net Output Winning entry for ActivityNet Classification task Wang, Limin, et al. "Towards good practices for very deep two-stream convnets." arXiv preprint arXiv:1507.02159 (2015).
- 72. Conclusions 72 Best results were obtained for sport categories, due to the pretraining of C3D with the Sports-1M dataset
- 73. Future Work: E2E Training 73 Training the whole pipeline end-to-end would reduce the bias towards sport categories
- 74. Future Work: Attention Models 74 Temporal Attention Filters Neural Network
- 77. “Thank you for your attention 77
- 78. 78 Questions?
- 85. AP and Video Appearance Correlation 85
- 86. AP and Video Appearance Correlation 86
- 88. Preparing Data 88 Sequence of Video Vector Features Sequence of Activities time
- 92. Gathering Audio Features 92 16-Frame Clip 10 ms MFCC Features t 10 ms MFCC Features 10 ms MFCC Features 10 ms MFCC Features 10 ms MFCC Features 10 ms MFCC Features 16-Frame Clip Spectral Features … … …
- 93. Gathering Audio Features 93 16-Frame Clip mean MFCC Features t std MFCC Features 16-Frame Clip Spectral Features … … … mean MFCC Features std MFCC Features
- 94. Gathering Audio Features 94 16-Frame Clip mean MFCC Features t std MFCC Features 16-Frame Clip Spectral Features … … … mean MFCC Features std MFCC Features Spectral Features
- 96. Convolutional Neural Network 96 Pooling Layer