SlideShare a Scribd company logo
Convolutional Patch Representations for Image
Retrieval: an Unsupervised Approach
29th Mar 2016
Original slides by Eva Mohedano
Insight Centre for Data Analytics (Dublin City University
Mattis Paulin, Julien Mairal, Matthijs Douze, Zaid Harchaoui, Florent Perronnin, Cordelia Schmidt
Overview
Published ICCV 2015 (A.K.A. Local Convolutional Features With Unsupervised
Training for Image Retrieval)
Deep Convolutional Architecture to produce patch-level descriptors
• Unsupervised framework
• Comparison in patch and retrieval datasets
• “RomePatches” dataset
Related Work
• Shallow patch descriptors
• Deep learning for image retrieval
• Deep patch descriptors
Related Work
• Shallow patch descriptors
SIFT – Scale-Invariant Feature Transform
- stereo matching
- retrieval
- classification
SURF, BRIEF, LIOP, (…)
Hand crafted → Relatively small number of parameters.
Note: A patch is an
image region extracted
from an image.
Related Work
• Deep learning for image retrieval
CNN learned on a sufficiently large labeled dataset (ImageNet) generates intermediate layers that
can be used as image descriptors.
Those descriptors work for a wide variety of tasks, including image retrieval
Related Work
• Deep learning for image retrieval
source image: http://pubs.sciepub.com/ajme/2/7/9/
Related Work
• Deep learning for image retrieval
source image: http://pubs.sciepub.com/ajme/2/7/9/
Fully connected layers → Global Image Descriptors
● Compact representation
● lack of geometric invariance
Below state-of-the art in image
retrieval
Compute at different scales
(Babenko, Razavian)
Related Work
• Deep learning for image retrieval
source image: http://pubs.sciepub.com/ajme/2/7/9/
Convolutional layers
Related Work
• Deep patch descriptors
3 different kind of supervision:
1. Category labels of ImageNet. [Long et al, 2014]
2. Surrogate patch labels: Each class is a given patch under different transformations [Fischer et al, 2014]
3. Matching/non-matching pairs. [Simo-Serra et al, 2015]
Works focussed in patch-level metrics, not image retrieval.
All approaches requiered some kind of supervision.
Image Retrieval Pipeline
• Interest point detection
Hessian-Affine detector.
Rotation invariance.
• Interest point description
Feature representation in a Euclidean space
• Patch Matching
VLAD encoding.
Power normalization with exponent 0.5 + L2-norm.
Image Retrieval Pipeline
• Interest point detection
Hessian-Affine detector.
Rotation invariance.
• Interest point description
Feature representation in a Euclidean space
• Patch Matching
VLAD encoding.
Power normalization with exponent 0.5 + L2-norm.
Convolutional Descriptors
Patch size = 51x51 – Optimal for SIFT on Oxford dataset.
CNN extended to retrieval by:
• Encoding local descriptors with model trained with an unrelated
classification task
• Devising a surrogate classification problem that is as related as
possible to image retrieval:
• Using unsupervised learning: Convolutional Kernel Network
Convolutional Descriptors
• Using unsupervised learning: Convolutional Kernel Network
Feature representation based in a kernel (feature) map -- Data independent
Convolutional Descriptors
• Using unsupervised learning: Convolutional Kernel Network
Projection in Hilbert space
Explicit kernel map can be computed to approximate it for computational efficiency.
- Sub-sample of patches
- Stochastic Gradient Optimization
Convolutional Descriptors
• Using unsupervised learning: Convolutional Kernel Network
4 possible inputs
From left to right: CKN-raw, CKN-mean subs, CKN-white (mean subs + PCA-whitening), CKN-grad
(fully invariant to color)
Only CKN-raw, CKN-white and CKN-grad are evaluated.
Experiments
Datasets:
1. Rome Patches-Image
2. Oxford
3. UKbench and Holidays
CKN trained on 1M sub-patches. 300K iterations. Mini-batches size of 1000.
Experiments
Conclusions
• CKN offer similar and sometimes better performance than CNN in the
context of patch description.
• Good patch retrieval translates into good image retrieval.
• CKNs are orders of magnitude faster to train than CNNs (10 min vs 2-3 days
on a modern GPU)
• Fully unsupervised – no labels.
Resources
RomePatches+Code (Although code is not accessible!)
Discriminative Unsupervised Feature Learning with Exemplar Convolutional
Neural Networks
- Code with augmentations in matlab
- Code for training models.
- Models already trained :-)
Triplet’s net + Code !!
- Greyscale local patches of 32x32. Tested in matching datasets
Ad

Recommended

Deep image retrieval - learning global representations for image search - ub ...
Deep image retrieval - learning global representations for image search - ub ...
Universitat de Barcelona
 
Deep image retrieval learning global representations for image search
Deep image retrieval learning global representations for image search
Universitat Politècnica de Catalunya
 
Class Weighted Convolutional Features for Image Retrieval
Class Weighted Convolutional Features for Image Retrieval
Universitat Politècnica de Catalunya
 
Convolutional Features for Instance Search
Convolutional Features for Instance Search
Universitat Politècnica de Catalunya
 
Deep Learning for Computer Vision: Image Retrieval (UPC 2016)
Deep Learning for Computer Vision: Image Retrieval (UPC 2016)
Universitat Politècnica de Catalunya
 
Semantic Segmentation on Satellite Imagery
Semantic Segmentation on Satellite Imagery
RAHUL BHOJWANI
 
object detection paper review
object detection paper review
Yoonho Na
 
Content-Based Image Retrieval (D2L6 Insight@DCU Machine Learning Workshop 2017)
Content-Based Image Retrieval (D2L6 Insight@DCU Machine Learning Workshop 2017)
Universitat Politècnica de Catalunya
 
Object Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning Framework
Nader Karimi
 
Mask-RCNN for Instance Segmentation
Mask-RCNN for Instance Segmentation
Dat Nguyen
 
R-FCN : object detection via region-based fully convolutional networks
R-FCN : object detection via region-based fully convolutional networks
Entrepreneur / Startup
 
Object detection - RCNNs vs Retinanet
Object detection - RCNNs vs Retinanet
Rishabh Indoria
 
Fast Non-Uniform Filtering with Symmetric Weighted Integral Images
Fast Non-Uniform Filtering with Symmetric Weighted Integral Images
davidmarimon
 
Improving access to satellite imagery with Cloud computing
Improving access to satellite imagery with Cloud computing
RAHUL BHOJWANI
 
Semantic segmentation with Convolutional Neural Network Approaches
Semantic segmentation with Convolutional Neural Network Approaches
UMBC
 
Deep learning based object detection basics
Deep learning based object detection basics
Brodmann17
 
DNR - Auto deep lab paper review ppt
DNR - Auto deep lab paper review ppt
taeseon ryu
 
Objects as points (CenterNet) review [CDM]
Objects as points (CenterNet) review [CDM]
Dongmin Choi
 
Shai Avidan's Support vector tracking and ensemble tracking
Shai Avidan's Support vector tracking and ensemble tracking
wolf
 
Deep Learning for Computer Vision: Segmentation (UPC 2016)
Deep Learning for Computer Vision: Segmentation (UPC 2016)
Universitat Politècnica de Catalunya
 
Semantic Mapping of Road Scenes
Semantic Mapping of Road Scenes
Sunando Sengupta
 
Object Pose Estimation
Object Pose Estimation
Arithmer Inc.
 
Accelerated Logistic Regression on GPU(s)
Accelerated Logistic Regression on GPU(s)
RAHUL BHOJWANI
 
Object Segmentation (D2L7 Insight@DCU Machine Learning Workshop 2017)
Object Segmentation (D2L7 Insight@DCU Machine Learning Workshop 2017)
Universitat Politècnica de Catalunya
 
Region-oriented Convolutional Networks for Object Retrieval
Region-oriented Convolutional Networks for Object Retrieval
Universitat Politècnica de Catalunya
 
How much position information do convolutional neural networks encode? review...
How much position information do convolutional neural networks encode? review...
Dongmin Choi
 
Deformable DETR Review [CDM]
Deformable DETR Review [CDM]
Dongmin Choi
 
Review: Incremental Few-shot Instance Segmentation [CDM]
Review: Incremental Few-shot Instance Segmentation [CDM]
Dongmin Choi
 
On-the-fly Visual Category Search in Web-scale Image Collections
On-the-fly Visual Category Search in Web-scale Image Collections
Ken Chatfield
 
Content-based Image Retrieval - Eva Mohedano - UPC Barcelona 2018
Content-based Image Retrieval - Eva Mohedano - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 

More Related Content

What's hot (20)

Object Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning Framework
Nader Karimi
 
Mask-RCNN for Instance Segmentation
Mask-RCNN for Instance Segmentation
Dat Nguyen
 
R-FCN : object detection via region-based fully convolutional networks
R-FCN : object detection via region-based fully convolutional networks
Entrepreneur / Startup
 
Object detection - RCNNs vs Retinanet
Object detection - RCNNs vs Retinanet
Rishabh Indoria
 
Fast Non-Uniform Filtering with Symmetric Weighted Integral Images
Fast Non-Uniform Filtering with Symmetric Weighted Integral Images
davidmarimon
 
Improving access to satellite imagery with Cloud computing
Improving access to satellite imagery with Cloud computing
RAHUL BHOJWANI
 
Semantic segmentation with Convolutional Neural Network Approaches
Semantic segmentation with Convolutional Neural Network Approaches
UMBC
 
Deep learning based object detection basics
Deep learning based object detection basics
Brodmann17
 
DNR - Auto deep lab paper review ppt
DNR - Auto deep lab paper review ppt
taeseon ryu
 
Objects as points (CenterNet) review [CDM]
Objects as points (CenterNet) review [CDM]
Dongmin Choi
 
Shai Avidan's Support vector tracking and ensemble tracking
Shai Avidan's Support vector tracking and ensemble tracking
wolf
 
Deep Learning for Computer Vision: Segmentation (UPC 2016)
Deep Learning for Computer Vision: Segmentation (UPC 2016)
Universitat Politècnica de Catalunya
 
Semantic Mapping of Road Scenes
Semantic Mapping of Road Scenes
Sunando Sengupta
 
Object Pose Estimation
Object Pose Estimation
Arithmer Inc.
 
Accelerated Logistic Regression on GPU(s)
Accelerated Logistic Regression on GPU(s)
RAHUL BHOJWANI
 
Object Segmentation (D2L7 Insight@DCU Machine Learning Workshop 2017)
Object Segmentation (D2L7 Insight@DCU Machine Learning Workshop 2017)
Universitat Politècnica de Catalunya
 
Region-oriented Convolutional Networks for Object Retrieval
Region-oriented Convolutional Networks for Object Retrieval
Universitat Politècnica de Catalunya
 
How much position information do convolutional neural networks encode? review...
How much position information do convolutional neural networks encode? review...
Dongmin Choi
 
Deformable DETR Review [CDM]
Deformable DETR Review [CDM]
Dongmin Choi
 
Review: Incremental Few-shot Instance Segmentation [CDM]
Review: Incremental Few-shot Instance Segmentation [CDM]
Dongmin Choi
 
Object Detection Using R-CNN Deep Learning Framework
Object Detection Using R-CNN Deep Learning Framework
Nader Karimi
 
Mask-RCNN for Instance Segmentation
Mask-RCNN for Instance Segmentation
Dat Nguyen
 
R-FCN : object detection via region-based fully convolutional networks
R-FCN : object detection via region-based fully convolutional networks
Entrepreneur / Startup
 
Object detection - RCNNs vs Retinanet
Object detection - RCNNs vs Retinanet
Rishabh Indoria
 
Fast Non-Uniform Filtering with Symmetric Weighted Integral Images
Fast Non-Uniform Filtering with Symmetric Weighted Integral Images
davidmarimon
 
Improving access to satellite imagery with Cloud computing
Improving access to satellite imagery with Cloud computing
RAHUL BHOJWANI
 
Semantic segmentation with Convolutional Neural Network Approaches
Semantic segmentation with Convolutional Neural Network Approaches
UMBC
 
Deep learning based object detection basics
Deep learning based object detection basics
Brodmann17
 
DNR - Auto deep lab paper review ppt
DNR - Auto deep lab paper review ppt
taeseon ryu
 
Objects as points (CenterNet) review [CDM]
Objects as points (CenterNet) review [CDM]
Dongmin Choi
 
Shai Avidan's Support vector tracking and ensemble tracking
Shai Avidan's Support vector tracking and ensemble tracking
wolf
 
Semantic Mapping of Road Scenes
Semantic Mapping of Road Scenes
Sunando Sengupta
 
Object Pose Estimation
Object Pose Estimation
Arithmer Inc.
 
Accelerated Logistic Regression on GPU(s)
Accelerated Logistic Regression on GPU(s)
RAHUL BHOJWANI
 
Object Segmentation (D2L7 Insight@DCU Machine Learning Workshop 2017)
Object Segmentation (D2L7 Insight@DCU Machine Learning Workshop 2017)
Universitat Politècnica de Catalunya
 
How much position information do convolutional neural networks encode? review...
How much position information do convolutional neural networks encode? review...
Dongmin Choi
 
Deformable DETR Review [CDM]
Deformable DETR Review [CDM]
Dongmin Choi
 
Review: Incremental Few-shot Instance Segmentation [CDM]
Review: Incremental Few-shot Instance Segmentation [CDM]
Dongmin Choi
 

Similar to Convolutional Patch Representations for Image Retrieval An unsupervised approach (20)

On-the-fly Visual Category Search in Web-scale Image Collections
On-the-fly Visual Category Search in Web-scale Image Collections
Ken Chatfield
 
Content-based Image Retrieval - Eva Mohedano - UPC Barcelona 2018
Content-based Image Retrieval - Eva Mohedano - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
conv_nets.pptx
conv_nets.pptx
ssuser80a05c
 
Image Retrieval (D4L5 2017 UPC Deep Learning for Computer Vision)
Image Retrieval (D4L5 2017 UPC Deep Learning for Computer Vision)
Universitat Politècnica de Catalunya
 
Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016
Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016
Universitat Politècnica de Catalunya
 
convnets.pptx
convnets.pptx
MohamedAliHabib3
 
Computer vision for transportation
Computer vision for transportation
Wanjin Yu
 
Lecture 29 Convolutional Neural Networks - Computer Vision Spring2015
Lecture 29 Convolutional Neural Networks - Computer Vision Spring2015
Jia-Bin Huang
 
Faire de la reconnaissance d'images avec le Deep Learning - Cristina & Pierre...
Faire de la reconnaissance d'images avec le Deep Learning - Cristina & Pierre...
Jedha Bootcamp
 
Overview of Convolutional Neural Networks
Overview of Convolutional Neural Networks
ananth
 
L7_finetuning on tamil technologies.pptx
L7_finetuning on tamil technologies.pptx
Meganath7
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
UMBC
 
Cs231n 2017 lecture12 Visualizing and Understanding
Cs231n 2017 lecture12 Visualizing and Understanding
Yanbin Kong
 
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Universitat Politècnica de Catalunya
 
Batik image retrieval using convolutional neural network
Batik image retrieval using convolutional neural network
TELKOMNIKA JOURNAL
 
CompVis 101 - Computer Vision Bootcamp - GDGoC IBA
CompVis 101 - Computer Vision Bootcamp - GDGoC IBA
HibaMallick1
 
Mnist report ppt
Mnist report ppt
RaghunandanJairam
 
1 introduction.pptx
1 introduction.pptx
mustafa sarac
 
Introduction to computer vision
Introduction to computer vision
Marcin Jedyk
 
Computer Vision Bootcamp: First Worshop
Computer Vision Bootcamp: First Worshop
MohammedArbi
 
On-the-fly Visual Category Search in Web-scale Image Collections
On-the-fly Visual Category Search in Web-scale Image Collections
Ken Chatfield
 
Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016
Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016
Universitat Politècnica de Catalunya
 
Computer vision for transportation
Computer vision for transportation
Wanjin Yu
 
Lecture 29 Convolutional Neural Networks - Computer Vision Spring2015
Lecture 29 Convolutional Neural Networks - Computer Vision Spring2015
Jia-Bin Huang
 
Faire de la reconnaissance d'images avec le Deep Learning - Cristina & Pierre...
Faire de la reconnaissance d'images avec le Deep Learning - Cristina & Pierre...
Jedha Bootcamp
 
Overview of Convolutional Neural Networks
Overview of Convolutional Neural Networks
ananth
 
L7_finetuning on tamil technologies.pptx
L7_finetuning on tamil technologies.pptx
Meganath7
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
UMBC
 
Cs231n 2017 lecture12 Visualizing and Understanding
Cs231n 2017 lecture12 Visualizing and Understanding
Yanbin Kong
 
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Interpretability of Convolutional Neural Networks - Xavier Giro - UPC Barcelo...
Universitat Politècnica de Catalunya
 
Batik image retrieval using convolutional neural network
Batik image retrieval using convolutional neural network
TELKOMNIKA JOURNAL
 
CompVis 101 - Computer Vision Bootcamp - GDGoC IBA
CompVis 101 - Computer Vision Bootcamp - GDGoC IBA
HibaMallick1
 
Introduction to computer vision
Introduction to computer vision
Marcin Jedyk
 
Computer Vision Bootcamp: First Worshop
Computer Vision Bootcamp: First Worshop
MohammedArbi
 
Ad

Recently uploaded (20)

Lesson-3_Program-Outcomes-and-Student-Learning-Outcomes_For-Students.pdf
Lesson-3_Program-Outcomes-and-Student-Learning-Outcomes_For-Students.pdf
SarahMaeDuallo
 
presentation4.pdf Intro to mcmc methodss
presentation4.pdf Intro to mcmc methodss
SergeyTsygankov6
 
一比一原版(TUC毕业证书)开姆尼茨工业大学毕业证如何办理
一比一原版(TUC毕业证书)开姆尼茨工业大学毕业证如何办理
taqyed
 
Communication_Skills_Class10_Visual.pptx
Communication_Skills_Class10_Visual.pptx
namanrastogi70555
 
NASA ESE Study Results v4 05.29.2020.pptx
NASA ESE Study Results v4 05.29.2020.pptx
CiroAlejandroCamacho
 
最新版美国加利福尼亚大学旧金山法学院毕业证(UCLawSF毕业证书)定制
最新版美国加利福尼亚大学旧金山法学院毕业证(UCLawSF毕业证书)定制
taqyea
 
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints: A...
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints: A...
Mahmoud Shoush
 
Indigo_Airlines_Strategy_Presentation.pptx
Indigo_Airlines_Strategy_Presentation.pptx
mukeshpurohit991
 
美国毕业证范本中华盛顿大学学位证书CWU学生卡购买
美国毕业证范本中华盛顿大学学位证书CWU学生卡购买
Taqyea
 
Microsoft Power BI - Advanced Certificate for Business Intelligence using Pow...
Microsoft Power BI - Advanced Certificate for Business Intelligence using Pow...
Prasenjit Debnath
 
最新版美国约翰霍普金斯大学毕业证(JHU毕业证书)原版定制
最新版美国约翰霍普金斯大学毕业证(JHU毕业证书)原版定制
Taqyea
 
BCG-Executive-Perspectives-CEOs-Guide-to-Maximizing-Value-from-AI-EP0-3July20...
BCG-Executive-Perspectives-CEOs-Guide-to-Maximizing-Value-from-AI-EP0-3July20...
benediktnetzer1
 
Model Evaluation & Visualisation part of a series of intro modules for data ...
Model Evaluation & Visualisation part of a series of intro modules for data ...
brandonlee626749
 
Boost Business Efficiency with Professional Data Entry Services
Boost Business Efficiency with Professional Data Entry Services
eloiacs eloiacs
 
Flextronics Employee Safety Data-Project-2.pptx
Flextronics Employee Safety Data-Project-2.pptx
kilarihemadri
 
Presentation by Tariq & Mohammed (1).pptx
Presentation by Tariq & Mohammed (1).pptx
AbooddSandoqaa
 
624753984-Annex-A3-RPMS-Tool-for-Proficient-Teachers-SY-2024-2025.pdf
624753984-Annex-A3-RPMS-Tool-for-Proficient-Teachers-SY-2024-2025.pdf
CristineGraceAcuyan
 
Residential Zone 4 for industrial village
Residential Zone 4 for industrial village
MdYasinArafat13
 
ppt somu_Jarvis_AI_Assistant_presen.pptx
ppt somu_Jarvis_AI_Assistant_presen.pptx
MohammedumarFarhan
 
All the DataOps, all the paradigms .
All the DataOps, all the paradigms .
Lars Albertsson
 
Lesson-3_Program-Outcomes-and-Student-Learning-Outcomes_For-Students.pdf
Lesson-3_Program-Outcomes-and-Student-Learning-Outcomes_For-Students.pdf
SarahMaeDuallo
 
presentation4.pdf Intro to mcmc methodss
presentation4.pdf Intro to mcmc methodss
SergeyTsygankov6
 
一比一原版(TUC毕业证书)开姆尼茨工业大学毕业证如何办理
一比一原版(TUC毕业证书)开姆尼茨工业大学毕业证如何办理
taqyed
 
Communication_Skills_Class10_Visual.pptx
Communication_Skills_Class10_Visual.pptx
namanrastogi70555
 
NASA ESE Study Results v4 05.29.2020.pptx
NASA ESE Study Results v4 05.29.2020.pptx
CiroAlejandroCamacho
 
最新版美国加利福尼亚大学旧金山法学院毕业证(UCLawSF毕业证书)定制
最新版美国加利福尼亚大学旧金山法学院毕业证(UCLawSF毕业证书)定制
taqyea
 
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints: A...
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints: A...
Mahmoud Shoush
 
Indigo_Airlines_Strategy_Presentation.pptx
Indigo_Airlines_Strategy_Presentation.pptx
mukeshpurohit991
 
美国毕业证范本中华盛顿大学学位证书CWU学生卡购买
美国毕业证范本中华盛顿大学学位证书CWU学生卡购买
Taqyea
 
Microsoft Power BI - Advanced Certificate for Business Intelligence using Pow...
Microsoft Power BI - Advanced Certificate for Business Intelligence using Pow...
Prasenjit Debnath
 
最新版美国约翰霍普金斯大学毕业证(JHU毕业证书)原版定制
最新版美国约翰霍普金斯大学毕业证(JHU毕业证书)原版定制
Taqyea
 
BCG-Executive-Perspectives-CEOs-Guide-to-Maximizing-Value-from-AI-EP0-3July20...
BCG-Executive-Perspectives-CEOs-Guide-to-Maximizing-Value-from-AI-EP0-3July20...
benediktnetzer1
 
Model Evaluation & Visualisation part of a series of intro modules for data ...
Model Evaluation & Visualisation part of a series of intro modules for data ...
brandonlee626749
 
Boost Business Efficiency with Professional Data Entry Services
Boost Business Efficiency with Professional Data Entry Services
eloiacs eloiacs
 
Flextronics Employee Safety Data-Project-2.pptx
Flextronics Employee Safety Data-Project-2.pptx
kilarihemadri
 
Presentation by Tariq & Mohammed (1).pptx
Presentation by Tariq & Mohammed (1).pptx
AbooddSandoqaa
 
624753984-Annex-A3-RPMS-Tool-for-Proficient-Teachers-SY-2024-2025.pdf
624753984-Annex-A3-RPMS-Tool-for-Proficient-Teachers-SY-2024-2025.pdf
CristineGraceAcuyan
 
Residential Zone 4 for industrial village
Residential Zone 4 for industrial village
MdYasinArafat13
 
ppt somu_Jarvis_AI_Assistant_presen.pptx
ppt somu_Jarvis_AI_Assistant_presen.pptx
MohammedumarFarhan
 
All the DataOps, all the paradigms .
All the DataOps, all the paradigms .
Lars Albertsson
 
Ad

Convolutional Patch Representations for Image Retrieval An unsupervised approach

  • 1. Convolutional Patch Representations for Image Retrieval: an Unsupervised Approach 29th Mar 2016 Original slides by Eva Mohedano Insight Centre for Data Analytics (Dublin City University Mattis Paulin, Julien Mairal, Matthijs Douze, Zaid Harchaoui, Florent Perronnin, Cordelia Schmidt
  • 2. Overview Published ICCV 2015 (A.K.A. Local Convolutional Features With Unsupervised Training for Image Retrieval) Deep Convolutional Architecture to produce patch-level descriptors • Unsupervised framework • Comparison in patch and retrieval datasets • “RomePatches” dataset
  • 3. Related Work • Shallow patch descriptors • Deep learning for image retrieval • Deep patch descriptors
  • 4. Related Work • Shallow patch descriptors SIFT – Scale-Invariant Feature Transform - stereo matching - retrieval - classification SURF, BRIEF, LIOP, (…) Hand crafted → Relatively small number of parameters. Note: A patch is an image region extracted from an image.
  • 5. Related Work • Deep learning for image retrieval CNN learned on a sufficiently large labeled dataset (ImageNet) generates intermediate layers that can be used as image descriptors. Those descriptors work for a wide variety of tasks, including image retrieval
  • 6. Related Work • Deep learning for image retrieval source image: http://pubs.sciepub.com/ajme/2/7/9/
  • 7. Related Work • Deep learning for image retrieval source image: http://pubs.sciepub.com/ajme/2/7/9/ Fully connected layers → Global Image Descriptors ● Compact representation ● lack of geometric invariance Below state-of-the art in image retrieval Compute at different scales (Babenko, Razavian)
  • 8. Related Work • Deep learning for image retrieval source image: http://pubs.sciepub.com/ajme/2/7/9/ Convolutional layers
  • 9. Related Work • Deep patch descriptors 3 different kind of supervision: 1. Category labels of ImageNet. [Long et al, 2014] 2. Surrogate patch labels: Each class is a given patch under different transformations [Fischer et al, 2014] 3. Matching/non-matching pairs. [Simo-Serra et al, 2015] Works focussed in patch-level metrics, not image retrieval. All approaches requiered some kind of supervision.
  • 10. Image Retrieval Pipeline • Interest point detection Hessian-Affine detector. Rotation invariance. • Interest point description Feature representation in a Euclidean space • Patch Matching VLAD encoding. Power normalization with exponent 0.5 + L2-norm.
  • 11. Image Retrieval Pipeline • Interest point detection Hessian-Affine detector. Rotation invariance. • Interest point description Feature representation in a Euclidean space • Patch Matching VLAD encoding. Power normalization with exponent 0.5 + L2-norm.
  • 12. Convolutional Descriptors Patch size = 51x51 – Optimal for SIFT on Oxford dataset. CNN extended to retrieval by: • Encoding local descriptors with model trained with an unrelated classification task • Devising a surrogate classification problem that is as related as possible to image retrieval: • Using unsupervised learning: Convolutional Kernel Network
  • 13. Convolutional Descriptors • Using unsupervised learning: Convolutional Kernel Network Feature representation based in a kernel (feature) map -- Data independent
  • 14. Convolutional Descriptors • Using unsupervised learning: Convolutional Kernel Network Projection in Hilbert space Explicit kernel map can be computed to approximate it for computational efficiency. - Sub-sample of patches - Stochastic Gradient Optimization
  • 15. Convolutional Descriptors • Using unsupervised learning: Convolutional Kernel Network 4 possible inputs From left to right: CKN-raw, CKN-mean subs, CKN-white (mean subs + PCA-whitening), CKN-grad (fully invariant to color) Only CKN-raw, CKN-white and CKN-grad are evaluated.
  • 16. Experiments Datasets: 1. Rome Patches-Image 2. Oxford 3. UKbench and Holidays CKN trained on 1M sub-patches. 300K iterations. Mini-batches size of 1000.
  • 18. Conclusions • CKN offer similar and sometimes better performance than CNN in the context of patch description. • Good patch retrieval translates into good image retrieval. • CKNs are orders of magnitude faster to train than CNNs (10 min vs 2-3 days on a modern GPU) • Fully unsupervised – no labels.
  • 19. Resources RomePatches+Code (Although code is not accessible!) Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks - Code with augmentations in matlab - Code for training models. - Models already trained :-) Triplet’s net + Code !! - Greyscale local patches of 32x32. Tested in matching datasets