PointNet
1 like•3,657 views
The document discusses the application of deep learning techniques to 3D point cloud data, emphasizing its significance in fields such as augmented reality and autonomous driving. It highlights ongoing research efforts, such as workshops aimed at fostering interdisciplinary collaboration to improve 3D data processing, and introduces PointNet, a novel architecture for classification and segmentation of unordered point clouds. The document also outlines the architectural components and performance metrics of PointNet, including pooling operations and potential applications in various domains.
![PointNet Modifications Architecture #1: Uncertainty estimation?
https://arxiv.org/pdf/1703.04977.pdf
http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html
[in classification
pipeline only] not in
segmentation part](https://image.slidesharecdn.com/3dv2017initreport-170503031811/85/PointNet-11-320.jpg)
PointNet
- 2. Implementation Initial ‘deep learning’ idea .XYZ point cloud better than the reconstructed .obj file for automatic segmentation due to higher resolution InputPointCloud 3D CAD MODEL No need to have planar surfaces Sampled too densely www.outsource3dcadmodeling.com 2DCAD MODEL Straightforward from 3D to 2D cadcrowd.com RECONSTRUCT 3D “Deep Learning” 3DSemantic Segmentation frompointcloud / reconstructed mesh youtube.com/watch?v=cGuoyNY54kU arxiv.org/1608.04236 Primitive-based deep learning segmentation The order between semantic segmentation and reconstruction could be swapped
- 3. NIPS 2016: 3D Workshop very early still for point cloud pipelines compared to “ordered images” Deep learning is proven to be a powerful tool to build models for language (one-dimensional) and image (two-dimensional) understanding. Tremendous efforts have been devoted to these areas, however, it is still at the early stage to apply deep learning to 3D data, despite their great research values and broad real- world applications. In particular, existing methods poorly serve the three-dimensional data that drives a broad range of critical applications such as augmented reality, autonomous driving, graphics, robotics, medical imaging, neuroscience, and scientific simulations. These problems have drawn the attention of researchers in different fields such as neuroscience, computer vision, and graphics. The goal of this workshop is to foster interdisciplinary communication of researchers working on 3D data (Computer Vision and Computer Graphics) so that more attention of broader community can be drawn to 3D deep learning problems. Through those studies, new ideas and discoveries are expected to emerge, which can inspire advances in related fields. This workshop is composed of invited talks, oral presentations of outstanding submissions and a poster session to showcase the state-of-the-art results on the topic. In particular, a panel discussion among leading researchers in the field is planned, so as to provide a common playground for inspiring discussions and stimulating debates. The workshop will be held on Dec 9 at NIPS 2016 in Barcelona, Spain. http://3ddl.cs.princeton.edu/2016/ ORGANIZERS ● Fisher Yu - Princeton University ● Joseph Lim - Stanford University ● Matthew Fisher - Stanford University ● Qixing Huang - University of Texas at Austin ● Jianxiong Xiao - AutoX Inc. http://cvpr2017.thecvf.com/ In Honolulu, Hawaii “I am co-organizing the 2nd Workshop on Visual Understanding for Interaction in conjunction with CVPR 2017. Stay tuned for the details!” “Our workshop on Large- Scale Scene Under- standing Challenge is accepted by CVPR 2017. http://3ddl.cs.princeton.edu/2016/slides/su.pdf
- 4. PointNet Deep learning for point cloud classification and segmentation https://github.com/charlesq34/pointnethttps://arxiv.org/abs/1612.00593 Applications of PointNet. We propose a novel deep net architecture that consumes raw unordered point cloud (set of points) without voxelization or rendering. It is a unified architecture that learns both global and local point features, providing a simple, efficient and effective approach for a number of 3D recognition tasks
- 5. PointNet Architecture Our network has three key modules: 1) the max pooling layer as a symmetric function to aggregate information from all the points, 2) a local and global information combination structure, 3) and two joint alignment networks that align both input points and point features.
- 6. PointNet symmetry function #1: Multi-layer Perceptron http://iamaaditya.github.io/2016/03/one-by-one-convolution/ https://github.com/charlesq34/pointnet/blob/master/models/pointnet_cls_basic.py MLP implented as 1x1 2D convolution
- 7. PointNet symmetry function #2: Max Pooling https://www.quora.com/How-is-a-convolutional-neural-network-able-to-learn-invariant-features Jean Da Rolt, PhD, Computer Engineer, Professor: “After some thought, I do not believe that pooling operations are responsible for the translation invariant property in CNNs. I believe that invariance (at least to translation) is due to the convolution filters (not specifically the pooling) and due to the fully-connected layer. In conclusion, what makes a CNN invariant to object translation is the architecture of the neural network: the convolution filters and the fully-connected layer.” Artem Rozantsev, PhD Computer Vision & Machine Learning: “In addition to the previous answers, standard ConvNets are invariant only to transformationas that are present in the training data. However, there are works, which made a step towards training networks that are inherently invariant to transformations such as rotation and translation, for example” https://arxiv.org/abs/1703.00356, https://arxiv.org/abs/1612.04642 https://arxiv.org/abs/1512.07108 University College London Ecole Polytechnique Fedérale de Lausanne (EPFL), Lausanne, Switzerland Key to our approach is the use of a single symmetric function, max pooling. E ffectively the network learns a set of optimization functions/criteria that select interesting or informative points of the point cloud and encode the reason for their selection. The final fully connected layers of the network aggregate these learnt optimal values into the global descriptor for the entire shape as mentioned above (shape classification) or are used to predict per point labels (shape segmentation
- 8. PointNet Combination Structure (pg. 3) " Therefore, the model needs to be able to capture local structures from nearby points, and the combinatorial interactions among local structures" (pg. 4) " After computing the global point cloud feature vector, we feed it back to per point features by concatenating the global feature with each of the point features. Then we extract new per point features based on the combined point features - this time the per point feature is aware of both the local and global information" (pg. 8) "As discussed in Sec 4.2 (pg. 4), our network computes K (we take K = 1024 in this experiment) dimension point features for each point and aggregates all the *per-point local features* via a max pooling layer into a single K-dim vector, which forms the global shape descriptor." (pg. 13) "Normal Estimation In segmentation version of PointNet, local point features and global feature are concatenated in order to provide context to local points. However, it’s unclear whether the context is learnt through this concatenation. In this experiment, we validate our design by showing that our segmentation network can be trained to predict point normals, a local geometric property that is determined by a point’s neighborhood"
- 9. PointNet Alignment Network PointNet: (pg. 1) "Thus we can add a data-dependent spatial transformer network that attempts to canonicalize the data before the PointNet processes them, so as to further improve the results." PointNet: (pg. 4) However, transformation matrix in the feature space has much higher dimension than the spatial transform matrix (e.g. from 3 × 3 to 64 × 64), which greatly increase the difficulty of optimization. We therefore add a regularization term to our softmax training loss. We constraint the feature transformation matrix to be close to orthogonal matrix. We find that by adding the regularization term, the optimization becomes more stable and our model achieves better performance. In Fig 15 we see that performance grows as we increase the number of points however it saturates at around 1K points. The max layer size plays an important role, increasing the layer size from 64 to 1024 results in a 2−4% performance gain. It indicates that we need enough point feature functions to cover the 3D space in order to discriminate different shapes.
- 10. PointNet Modifications input data,increase dimensionality? PointNet: (pg. 1) "In the basic setting each point is represented by just its three coordinates (x, y, z). Additional dimensions may be added by computing normals and other local or global features." Data columns: x, y, z, red, green, blue, no normals Pointclouds canbe huge https://www.we-get-around.com/wegetaround- atlanta-our-blog/2015/10/cubicasa-creates- 2d-and-3d-floor-plans-for-matterport-photo graphers-from-3d-showcase-tours 6-dimensional inputdata With the x,y,z coordinates one obtains also R,G,B values (or CIE LAB colorspace) that are very useful in segmenting objects. 7-dimensional inputdata Normals could be obtained too if the camera position were known Eurographics Symposium on Geometry Processing 2016, Volume 35 (2016), Number 5 http://dx.doi.org/10.1111/cgf.12983 PointNet: (pg. 13)
- 11. PointNet Modifications Architecture #1: Uncertainty estimation? https://arxiv.org/pdf/1703.04977.pdf http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html [in classification pipeline only] not in segmentation part
- 12. PointNet Modifications Architecture #2: component variations? Nonlinearity Pooling Layer Normalization In order to make a model invariant to input permutation, the authors use max pooling as the simple symmetric function to aggregate the information from each point. [in classification[ All layers, except the last one, include ReLU and batch normalization. [in classification[ All layers, except the last one, include ReLU and batch normalization. http://arxiv.org/abs/1604.04112 “One possible future line of work is to embed the network in its entirety in the frequency domain. In models that employ Fourier transforms to compute convolutions, at every convolutional layer the input is FFT-ed and the elementwise multiplication output is then inverse FFT-ed. These back-andforth transformations are very computationally intensive, and as such it would be desirable to strictly remain in the frequency domain. However, the reason for these repeated transformations is the application of nonlinearities in the forward domain: if one were to propose a sensible nonlinearity in the frequency domain, this would spare us from the incessant domain switching.” Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. https://arxiv.org/abs/1602.07868 https://arxiv.org/abs/1605.09332 http://arxiv.org/abs/1512.07108
- 13. PointNet Modifications Architecture #3: Unsupervised/Semi-supervised extensions?