Mini-Batch Consistent Slot Set Encoder For Scalable Set Encoding
0 likes199 views
Most existing set encoding algorithms operate under the implicit assumption that all the set elements are accessible, and that there are ample computational and memory resources to load the set into memory during training and inference. However, both assumptions fail when the set is excessively large such that it is impossible to load all set elements into memory, or when data arrives in a stream. To tackle such practical challenges in large-scale set encoding, the general set-function constraints of permutation invariance and equivariance are not sufficient. We introduce a new property termed Mini-Batch Consistency (MBC) that is required for large scale mini-batch set encoding. Additionally, we present a scalable and efficient attention-based set encoding mechanism that is amenable to mini-batch processing of sets, and capable of updating set representations as data arrives. The proposed method adheres to the required symmetries of invariance and equivariance as well as maintaining MBC for any partition of the input set. We perform extensive experiments and show that our method is computationally efficient and results in rich set encoding representations for set-structured data.
1 of 12
Download to read offline
![Property 1 A function 𝒇: 𝟐𝑿 → 𝒀 acting on sets must be permutation invariant to the
order of objects in the set, i.e. for any permutation 𝝅:
𝒇 𝒙𝟏, … , 𝒙𝑴 = 𝒇 𝒙𝝅 𝟏 , … , 𝒙𝝅 𝑴 .
Exchangeability A distribution for a set of random variables 𝑿 = 𝒙𝒊 𝒊&𝟏
𝑴
is
exchangeable if for an permutation 𝝅:
𝒑 𝑿 = 𝒑 𝝅 𝑿 .
Property 2 A function 𝒇: 𝑿𝑴 → 𝒀𝑴 acting on sets is a permutation equivariant
function if permutation of the input instances permutes the output labels, i.e. for any
permutation 𝝅:
𝒇 𝒙𝝅 𝟏 , … , 𝒙𝝅 𝑴 = [𝒇𝝅 𝟏 𝒙 , … , 𝒇𝝅 𝑴 𝒙 ]
Permutation Invariance & Equivariance
Bloem-Reddy, Benjamin, and Yee Whye Teh. "Probabilistic Symmetries and Invariant Neural Networks." J. Mach. Learn. Res. 21 (2020): 90-1.
3](https://image.slidesharecdn.com/neurips2021mini-batchconsistentslotsetencoderforscalablesetencoding-211206042436/85/Mini-Batch-Consistent-Slot-Set-Encoder-For-Scalable-Set-Encoding-3-320.jpg)
![Hierarchical Slot Set Encoder
We can stack multiple Slot Set Encoders on top of each to obtain a hierarchy of slot
set encoders. This allows us to model higher-order interactions across slots.
𝑓 𝑋 = 𝑆𝑆𝐸 … 𝑆𝑆𝐸] 𝑆𝑆𝐸^ 𝑋
The resulting set encoding function 𝑓(𝑋) satisfies the MBC property as well as
Propositions 3 & 4.
Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615.
9](https://image.slidesharecdn.com/neurips2021mini-batchconsistentslotsetencoderforscalablesetencoding-211206042436/85/Mini-Batch-Consistent-Slot-Set-Encoder-For-Scalable-Set-Encoding-9-320.jpg)
Ad
Recommended
Cs229 notes7a
Cs229 notes7aVuTran231 The k-means clustering algorithm aims to group data points into k clusters based on their distances from initial cluster centroid points. It works by alternating between assigning each point to its nearest centroid, and updating the centroid locations to be the mean of their assigned points. This process monotonically decreases the distortion score measuring distances from points to centroids, and is guaranteed to converge, though possibly to local optima rather than the global minimum. Running it multiple times can help avoid bad initial results.
Skiena algorithm 2007 lecture18 application of dynamic programming
Skiena algorithm 2007 lecture18 application of dynamic programmingzukun The document summarizes a lecture on applications of dynamic programming. It provides examples of how to use dynamic programming to solve problems involving string breaking, high density bar code encoding, dividing work evenly among workers, and the traveling salesman problem. Dynamic programming can be applied when problems exhibit the principle of optimality and the problem space can be broken down into overlapping subproblems that are stored in a table to avoid recomputing solutions.
upgrade2013
upgrade2013Rhian Davies Compressive sensing is a method to reduce the amount of data collected from a signal while still allowing accurate reconstruction later. It works by taking measurements using a matrix that satisfies the restricted isometry property. Sparse signals can be recovered using optimization techniques that minimize the l1 norm. Compressive sensing can be applied to background subtraction in video analysis by compressing the background model and reconstructing the difference between current and background frames to identify foreground objects. Further work includes optimizing the measurement matrix and reconstruction method and adapting to varying sparsity in video frames.
Variational Auto Encoder and the Math Behind
Variational Auto Encoder and the Math BehindVarun Reddy The Math and internal workings of Variational Auto Encoders. It talks about Variational Inference and how VAEs take advantage of them to bypass having to calculate Marginal Probability distribution.
redes neuronales tipo Som
redes neuronales tipo SomESCOM This document discusses self-organizing neural networks, including Kohonen networks and Adaptive Resonance Theory (ART). It provides details on Kohonen networks such as their basic structure, learning algorithm using neighborhoods, and biological origins. ART is introduced as a way to address the stability-plasticity dilemma in neural networks. The key aspects of ART1 are summarized, including its orienting and attentional subsystems, short and long term memory representations, and learning algorithm using a vigilance test. Examples of a Kohonen network and ART1 network are also included to illustrate their operation.
Variational Autoencoders For Image Generation
Variational Autoencoders For Image GenerationJason Anderson Meetup: https://www.meetup.com/Cognitive-Computing-Enthusiasts/events/260580395/
Video: https://www.youtube.com/watch?v=fnULFOyNZn8
Blog: http://www.compthree.com/blog/autoencoder/
Code: https://github.com/compthree/variational-autoencoder
An autoencoder is a machine learning algorithm that represents unlabeled high-dimensional data as points in a low-dimensional space. A variational autoencoder (VAE) is an autoencoder that represents unlabeled high-dimensional data as low-dimensional probability distributions. In addition to data compression, the randomness of the VAE algorithm gives it a second powerful feature: the ability to generate new data similar to its training data. For example, a VAE trained on images of faces can generate a compelling image of a new "fake" face. It can also map new features onto input data, such as glasses or a mustache onto the image of a face that initially lacks these features. In this talk, we will survey VAE model designs that use deep learning, and we will implement a basic VAE in TensorFlow. We will also demonstrate the encoding and generative capabilities of VAEs and discuss their industry applications.
Self Organinising neural networks
Self Organinising neural networksESCOM This document discusses self-organizing neural networks, including Kohonen networks and Adaptive Resonance Theory (ART). Kohonen networks use competitive learning to form topological mappings between input and output layers. Neighboring units respond to similar inputs, and learning updates weights of both the winning unit and its neighbors. ART networks learn stable recognition codes in response to input sequences and address the stability-plasticity dilemma by resetting matches that fail a vigilance test.
IJCER (www.ijceronline.com) International Journal of computational Engineerin...
IJCER (www.ijceronline.com) International Journal of computational Engineerin...ijceronline The document discusses video compression using the Set Partitioning in Hierarchical Trees (SPIHT) algorithm and neural networks. It presents the principles of SPIHT coding and the backpropagation algorithm for neural networks. Various neural network training algorithms are tested for compressing video frames, including gradient descent with momentum and adaptive learning. The results show the compressed frames with different algorithms, and gradient descent with momentum and adaptive learning achieved the best compression ratio of 1.1737089:1 while maintaining image clarity.
Set Transfomer: A Framework for Attention-based Permutaion-Invariant Neural N...
Set Transfomer: A Framework for Attention-based Permutaion-Invariant Neural N...Thien Q. Tran An introduction of Set Transformer (ICML2019)
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek*,
Seungjin Choi, Yee Whye Teh
Parallel Machine Learning- DSGD and SystemML
Parallel Machine Learning- DSGD and SystemMLJanani C This document discusses parallel machine learning algorithms. It covers topics like parallel processing, supervised and unsupervised learning, linear regression for price prediction, gradient descent optimization, and matrix factorization. It also describes distributed stochastic gradient descent (DSGD) for parallelizing matrix factorization algorithms across multiple nodes. DSGD randomly permutes and partitions the data matrix before distributing the blocks to different nodes for parallel optimization.
Deep Learning con CNTK by Pablo Doval
Deep Learning con CNTK by Pablo DovalPlain Concepts The document is an agenda for a conference on "Intelligent Hospitals: Discovering Digital Health". The agenda includes keynote speeches and sessions on topics like storing and sharing medical images in Azure, applying artificial intelligence to healthcare using Cognitive Services, computer vision, Power BI and machine learning, chatbots, and using virtual and augmented reality in hospitals.
Score-Based Generative Modeling through Stochastic Differential Equations
Score-Based Generative Modeling through Stochastic Differential EquationsSangwoo Mo Lab seminar on "Score-Based Generative Modeling through Stochastic Differential Equations", ICLR 2021 under review
Deep Coder - Experimental Research Presentation
Deep Coder - Experimental Research PresentationDUONG Dinh Cuong Is this possible to train a machine writing the runnable code? What's the current research result today and what are the limitations?
Tensorflow London 13: Zbigniew Wojna 'Deep Learning for Big Scale 2D Imagery'
Tensorflow London 13: Zbigniew Wojna 'Deep Learning for Big Scale 2D Imagery'Seldon Speaker: Zbigniew Wojna, Deep Learning Researcher and founder of TensorFlight.Inc
Title: Architectures for big scale 2D imagery
Abstract: Zbigniew will present research he conducted during his Ph.D. at University College London and in collaboration with Google. His primary interest lays in the development of neural architectures for 2D imagery problems in big scale. He will present the recently published analysis of different upsampling methods in the decoder part of visual architectures, together with last week ongoing extension for GANs. Will discuss attention mechanism for text recognition and review for what kind of application it can be useful (automatically updating Google Maps based on Google Street View imagery). He will explain the idea behind inception and change in Inception-v3 to have it the best single model on ImageNet 2015 and how does it compare to Resnet architecture which was published 2 weeks after. Together with inception, will present his winning submission to MS COCO 2016 detection challenge and the extensive analysis of different models and backbone architectures inside. At the end will shortly review UCL effort working with 4096x4096 images at The Digital Mammography DREAM Challenge for breast cancer recognition, where they achieved 9th among 1375 teams worldwide and 2nd place in the community phase.
Bio: Zbigniew Wojna is deep learning researcher and founder of TensorFlight Inc. company providing instant remote commercial property inspection (for risk factors for reinsurance enterprises) based on satellite and street view type imagery. Zbigniew is currently in the final stage of his Ph.D. (already with more than 1000 citations) at the University College London under the supervision of Professor Iasonas Kokkinos and professor John Shawe-Taylor. His primary interest lies in finding and solving research problems around 2D machine vision applications usually in big scale. Zbigniew in his Ph.D. career spent most of the time working across different groups in DeepMind, Google Research, and Facebook Research. It includes DeepMind Health Team, Deep Learning Team for Google Maps in collaboration with Google Brain, Machine Perception with Kevin Murphy, Weak Localization Team with Vittorio Ferrari and Facebook AI Research Lab in Paris. His company TensorFlight Inc. was featured as top 2 AI startups among few hundreds by InnovatorsRace50 and closed seed funding last year.
Thanks to all TensorFlow London meetup organisers and supporters:
Seldon.io
Altoros
Rewired
Google Developers
Rise London
Towards Set Learning and Prediction - Laura Leal-Taixe - UPC Barcelona 2018
Towards Set Learning and Prediction - Laura Leal-Taixe - UPC Barcelona 2018Universitat Politècnica de Catalunya https://telecombcn-dl.github.io/2018-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.
International Journal of Image Processing (IJIP) Volume (4) Issue (1)
International Journal of Image Processing (IJIP) Volume (4) Issue (1)CSCJournals The paper proposes a two-level codebook generation algorithm to reduce mean squared error (MSE) for a given codebook size in vector quantization.
In the first level, x% of the codebook is generated using Kekre's Proportionate Error algorithm. The image is reconstructed and the error image is used to generate the remaining (100-x)% codebook in the second level.
Results show that the proposed two-level method reduces MSE compared to single-level codebook generation using Kekre's algorithm or LBG, for codebook sizes of 256, 512 and 1024. For example, allocating 67% of the 1024-sized codebook to the first level and the rest
A Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware DetectionIJCSIS Research Publications Deep learning algorithms have drawn the attention of researchers working in the field of computer vision, speech
recognition, malware detection, pattern recognition and natural language processing. In this paper, we present an overview of
deep learning techniques like Convolutional neural network, deep belief network, Autoencoder, Restricted Boltzmann machine
and recurrent neural network. With this, current work of deep learning algorithms on malware detection is shown with the
help of literature survey. Suggestions for future research are given with full justification. We also showed the experimental
analysis in order to show the importance of deep learning techniques.
Scaling Deep Learning with MXNet
Scaling Deep Learning with MXNetAI Frontiers Deep learning continues to push the state of the art in domains such as computer vision, natural language understanding and recommendation engines. One of the key reasons for this progress is the availability of highly flexible and developer friendly deep learning frameworks. During this workshop, members of the Amazon Machine Learning team will provide a short background on Deep Learning focusing on relevant application domains and an introduction to using the powerful and scalable Deep Learning framework, MXNet. At the end of this tutorial you’ll gain hands on experience targeting a variety of applications including computer vision and recommendation engines as well as exposure to how to use preconfigured Deep Learning AMIs and CloudFormation Templates to help speed your development.
IRJET- Handwritten Decimal Image Compression using Deep Stacked Autoencoder
IRJET- Handwritten Decimal Image Compression using Deep Stacked AutoencoderIRJET Journal This document proposes using a deep stacked autoencoder neural network for compressing handwritten decimal image data. It involves training multiple autoencoders in sequence to form a deep network that can compress the high-dimensional input images into lower-dimensional encoded representations while minimizing information loss. The autoencoders are trained one layer at a time using scaled conjugate gradient descent. Testing on the MNIST handwritten digits dataset showed the deep stacked autoencoder achieved compression by encoding the 400-dimensional input images down to a 25-dimensional representation while maintaining good reconstruction accuracy, as measured by minimizing the mean squared error at each layer.
Deep Generative Models I (DLAI D9L2 2017 UPC Deep Learning for Artificial Int...
Deep Generative Models I (DLAI D9L2 2017 UPC Deep Learning for Artificial Int...Universitat Politècnica de Catalunya This document provides an outline and introduction to deep generative models. It discusses what generative models are, their applications like image and speech generation/enhancement, and different types of generative models including PixelRNN/CNN, variational autoencoders, and generative adversarial networks. Variational autoencoders are explained in detail, covering how they introduce a restriction in the latent space z to generate new data points by sampling from the latent prior distribution.
Deep Learning workshop 2010: Deep Learning of Invariant Spatiotemporal Featur...
Deep Learning workshop 2010: Deep Learning of Invariant Spatiotemporal Featur...zukun The document discusses deep learning of invariant spatiotemporal features from video using a proposed spatiotemporal deep belief network model. It first provides background on restricted Boltzmann machines and convolutional RBMs for feature extraction from images. It then introduces the proposed spatiotemporal DBN model which extends convolutional RBMs to video by including temporal pooling layers. The model is trained greedily layer-wise using contrastive divergence. Experiments are conducted to measure the invariance of learned features and evaluate the model on action recognition and other tasks.
Introduction to Autoencoders
Introduction to AutoencodersYan Xu The document provides an introduction and overview of auto-encoders, including their architecture, learning and inference processes, and applications. It discusses how auto-encoders can learn hierarchical representations of data in an unsupervised manner by compressing the input into a code and then reconstructing the output from that code. Sparse auto-encoders and stacking multiple auto-encoders are also covered. The document uses handwritten digit recognition as an example application to illustrate these concepts.
DLD_WeightSharing_Slide
DLD_WeightSharing_SlideKang-Ho Lee - The document summarizes several techniques for compressing deep learning models, including weight sharing, pruning, quantization, and knowledge distillation.
- It then introduces the idea of applying weight sharing to compress deep learning models, describing how weights can be shared across layers or during quantization.
- The bulk of the document summarizes five recent papers that propose novel approaches for model compression using techniques like layer reuse, dynamic recursion, filter summarization, neural epitomes, and blueprint separable convolutions. Each technique aims to reduce model size or computations without significant loss in accuracy.
Restricted Boltzman Machine (RBM) presentation of fundamental theory
Restricted Boltzman Machine (RBM) presentation of fundamental theorySeongwon Hwang The document discusses restricted Boltzmann machines (RBMs), an type of neural network that can learn probability distributions over its input data. It explains that RBMs define an energy function over hidden and visible units, with no connections between units within the same group. This conditional independence allows efficient computation of conditional probabilities. RBMs are trained using maximum likelihood, minimizing the negative log-likelihood of the training data by gradient descent.
Deepcoder to Self-Code with Machine Learning
Deepcoder to Self-Code with Machine LearningIRJET Journal The document discusses DeepCoder, a machine learning system developed by Microsoft that is able to generate its own code by learning from existing code examples. DeepCoder is trained on a large corpus of programs and input/output examples to learn which code snippets are likely to work together to solve new problems. It can then search through code more efficiently than humans to assemble working programs from existing code blocks. While currently limited to simple 5 line programs, DeepCoder represents a significant improvement over previous program synthesis techniques and could eventually make programming accessible to non-coders. However, some media reports exaggerated DeepCoder's capabilities and inaccurately claimed it works by copying code directly from other software.
Deep learning simplified
Deep learning simplifiedLovelyn Rose The document discusses deep learning and artificial neural networks. It provides an agenda for topics covered, including gradient descent, backpropagation, activation functions, and examples of neural network architectures like convolutional neural networks. It explains concepts like how neural networks learn patterns from data using techniques like stochastic gradient descent to minimize loss functions. Deep learning requires large amounts of processing power and labeled training data. Common deep learning networks are used for tasks like image recognition, object detection, and time series analysis.
Intelligent Parallel Processing and Compound Image Compression
Intelligent Parallel Processing and Compound Image CompressionDR.P.S.JAGADEESH KUMAR This document discusses parallel processing and compound image compression techniques. It examines the computational complexity and quantitative optimization of various image compression algorithms like BTC, DCT, DWT, DTCWT, SPIHT and EZW. The performance is evaluated in terms of coding efficiency, memory usage, image quality and quantity. Block Truncation Coding and Discrete Cosine Transform compression methods are described in more detail.
Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Unce...
Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Unce...MLAI2 Numerous recent works utilize bi-Lipschitz regularization of neural network layers to preserve relative distances between data instances in the feature spaces of each layer. This distance sensitivity with respect to the data aids in tasks such as uncertainty calibration and out-of-distribution (OOD) detection. In previous works, features extracted with a distance sensitive model are used to construct feature covariance matrices which are used in deterministic uncertainty estimation or OOD detection. However, in cases where there is a distribution over tasks, these methods result in covariances which are sub-optimal, as they may not leverage all of the meta information which can be shared among tasks. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. Additionally, we propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution which is well calibrated under a distributional dataset shift.
Online Hyperparameter Meta-Learning with Hypergradient Distillation
Online Hyperparameter Meta-Learning with Hypergradient DistillationMLAI2 Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on two different meta-learning methods and three benchmark datasets.
More Related Content
Similar to Mini-Batch Consistent Slot Set Encoder For Scalable Set Encoding (20)
Set Transfomer: A Framework for Attention-based Permutaion-Invariant Neural N...
Set Transfomer: A Framework for Attention-based Permutaion-Invariant Neural N...Thien Q. Tran An introduction of Set Transformer (ICML2019)
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek*,
Seungjin Choi, Yee Whye Teh
Parallel Machine Learning- DSGD and SystemML
Parallel Machine Learning- DSGD and SystemMLJanani C This document discusses parallel machine learning algorithms. It covers topics like parallel processing, supervised and unsupervised learning, linear regression for price prediction, gradient descent optimization, and matrix factorization. It also describes distributed stochastic gradient descent (DSGD) for parallelizing matrix factorization algorithms across multiple nodes. DSGD randomly permutes and partitions the data matrix before distributing the blocks to different nodes for parallel optimization.
Deep Learning con CNTK by Pablo Doval
Deep Learning con CNTK by Pablo DovalPlain Concepts The document is an agenda for a conference on "Intelligent Hospitals: Discovering Digital Health". The agenda includes keynote speeches and sessions on topics like storing and sharing medical images in Azure, applying artificial intelligence to healthcare using Cognitive Services, computer vision, Power BI and machine learning, chatbots, and using virtual and augmented reality in hospitals.
Score-Based Generative Modeling through Stochastic Differential Equations
Score-Based Generative Modeling through Stochastic Differential EquationsSangwoo Mo Lab seminar on "Score-Based Generative Modeling through Stochastic Differential Equations", ICLR 2021 under review
Deep Coder - Experimental Research Presentation
Deep Coder - Experimental Research PresentationDUONG Dinh Cuong Is this possible to train a machine writing the runnable code? What's the current research result today and what are the limitations?
Tensorflow London 13: Zbigniew Wojna 'Deep Learning for Big Scale 2D Imagery'
Tensorflow London 13: Zbigniew Wojna 'Deep Learning for Big Scale 2D Imagery'Seldon Speaker: Zbigniew Wojna, Deep Learning Researcher and founder of TensorFlight.Inc
Title: Architectures for big scale 2D imagery
Abstract: Zbigniew will present research he conducted during his Ph.D. at University College London and in collaboration with Google. His primary interest lays in the development of neural architectures for 2D imagery problems in big scale. He will present the recently published analysis of different upsampling methods in the decoder part of visual architectures, together with last week ongoing extension for GANs. Will discuss attention mechanism for text recognition and review for what kind of application it can be useful (automatically updating Google Maps based on Google Street View imagery). He will explain the idea behind inception and change in Inception-v3 to have it the best single model on ImageNet 2015 and how does it compare to Resnet architecture which was published 2 weeks after. Together with inception, will present his winning submission to MS COCO 2016 detection challenge and the extensive analysis of different models and backbone architectures inside. At the end will shortly review UCL effort working with 4096x4096 images at The Digital Mammography DREAM Challenge for breast cancer recognition, where they achieved 9th among 1375 teams worldwide and 2nd place in the community phase.
Bio: Zbigniew Wojna is deep learning researcher and founder of TensorFlight Inc. company providing instant remote commercial property inspection (for risk factors for reinsurance enterprises) based on satellite and street view type imagery. Zbigniew is currently in the final stage of his Ph.D. (already with more than 1000 citations) at the University College London under the supervision of Professor Iasonas Kokkinos and professor John Shawe-Taylor. His primary interest lies in finding and solving research problems around 2D machine vision applications usually in big scale. Zbigniew in his Ph.D. career spent most of the time working across different groups in DeepMind, Google Research, and Facebook Research. It includes DeepMind Health Team, Deep Learning Team for Google Maps in collaboration with Google Brain, Machine Perception with Kevin Murphy, Weak Localization Team with Vittorio Ferrari and Facebook AI Research Lab in Paris. His company TensorFlight Inc. was featured as top 2 AI startups among few hundreds by InnovatorsRace50 and closed seed funding last year.
Thanks to all TensorFlow London meetup organisers and supporters:
Seldon.io
Altoros
Rewired
Google Developers
Rise London
Towards Set Learning and Prediction - Laura Leal-Taixe - UPC Barcelona 2018
Towards Set Learning and Prediction - Laura Leal-Taixe - UPC Barcelona 2018Universitat Politècnica de Catalunya https://telecombcn-dl.github.io/2018-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.
International Journal of Image Processing (IJIP) Volume (4) Issue (1)
International Journal of Image Processing (IJIP) Volume (4) Issue (1)CSCJournals The paper proposes a two-level codebook generation algorithm to reduce mean squared error (MSE) for a given codebook size in vector quantization.
In the first level, x% of the codebook is generated using Kekre's Proportionate Error algorithm. The image is reconstructed and the error image is used to generate the remaining (100-x)% codebook in the second level.
Results show that the proposed two-level method reduces MSE compared to single-level codebook generation using Kekre's algorithm or LBG, for codebook sizes of 256, 512 and 1024. For example, allocating 67% of the 1024-sized codebook to the first level and the rest
A Survey of Deep Learning Algorithms for Malware Detection
A Survey of Deep Learning Algorithms for Malware DetectionIJCSIS Research Publications Deep learning algorithms have drawn the attention of researchers working in the field of computer vision, speech
recognition, malware detection, pattern recognition and natural language processing. In this paper, we present an overview of
deep learning techniques like Convolutional neural network, deep belief network, Autoencoder, Restricted Boltzmann machine
and recurrent neural network. With this, current work of deep learning algorithms on malware detection is shown with the
help of literature survey. Suggestions for future research are given with full justification. We also showed the experimental
analysis in order to show the importance of deep learning techniques.
Scaling Deep Learning with MXNet
Scaling Deep Learning with MXNetAI Frontiers Deep learning continues to push the state of the art in domains such as computer vision, natural language understanding and recommendation engines. One of the key reasons for this progress is the availability of highly flexible and developer friendly deep learning frameworks. During this workshop, members of the Amazon Machine Learning team will provide a short background on Deep Learning focusing on relevant application domains and an introduction to using the powerful and scalable Deep Learning framework, MXNet. At the end of this tutorial you’ll gain hands on experience targeting a variety of applications including computer vision and recommendation engines as well as exposure to how to use preconfigured Deep Learning AMIs and CloudFormation Templates to help speed your development.
IRJET- Handwritten Decimal Image Compression using Deep Stacked Autoencoder
IRJET- Handwritten Decimal Image Compression using Deep Stacked AutoencoderIRJET Journal This document proposes using a deep stacked autoencoder neural network for compressing handwritten decimal image data. It involves training multiple autoencoders in sequence to form a deep network that can compress the high-dimensional input images into lower-dimensional encoded representations while minimizing information loss. The autoencoders are trained one layer at a time using scaled conjugate gradient descent. Testing on the MNIST handwritten digits dataset showed the deep stacked autoencoder achieved compression by encoding the 400-dimensional input images down to a 25-dimensional representation while maintaining good reconstruction accuracy, as measured by minimizing the mean squared error at each layer.
Deep Generative Models I (DLAI D9L2 2017 UPC Deep Learning for Artificial Int...
Deep Generative Models I (DLAI D9L2 2017 UPC Deep Learning for Artificial Int...Universitat Politècnica de Catalunya This document provides an outline and introduction to deep generative models. It discusses what generative models are, their applications like image and speech generation/enhancement, and different types of generative models including PixelRNN/CNN, variational autoencoders, and generative adversarial networks. Variational autoencoders are explained in detail, covering how they introduce a restriction in the latent space z to generate new data points by sampling from the latent prior distribution.
Deep Learning workshop 2010: Deep Learning of Invariant Spatiotemporal Featur...
Deep Learning workshop 2010: Deep Learning of Invariant Spatiotemporal Featur...zukun The document discusses deep learning of invariant spatiotemporal features from video using a proposed spatiotemporal deep belief network model. It first provides background on restricted Boltzmann machines and convolutional RBMs for feature extraction from images. It then introduces the proposed spatiotemporal DBN model which extends convolutional RBMs to video by including temporal pooling layers. The model is trained greedily layer-wise using contrastive divergence. Experiments are conducted to measure the invariance of learned features and evaluate the model on action recognition and other tasks.
Introduction to Autoencoders
Introduction to AutoencodersYan Xu The document provides an introduction and overview of auto-encoders, including their architecture, learning and inference processes, and applications. It discusses how auto-encoders can learn hierarchical representations of data in an unsupervised manner by compressing the input into a code and then reconstructing the output from that code. Sparse auto-encoders and stacking multiple auto-encoders are also covered. The document uses handwritten digit recognition as an example application to illustrate these concepts.
DLD_WeightSharing_Slide
DLD_WeightSharing_SlideKang-Ho Lee - The document summarizes several techniques for compressing deep learning models, including weight sharing, pruning, quantization, and knowledge distillation.
- It then introduces the idea of applying weight sharing to compress deep learning models, describing how weights can be shared across layers or during quantization.
- The bulk of the document summarizes five recent papers that propose novel approaches for model compression using techniques like layer reuse, dynamic recursion, filter summarization, neural epitomes, and blueprint separable convolutions. Each technique aims to reduce model size or computations without significant loss in accuracy.
Restricted Boltzman Machine (RBM) presentation of fundamental theory
Restricted Boltzman Machine (RBM) presentation of fundamental theorySeongwon Hwang The document discusses restricted Boltzmann machines (RBMs), an type of neural network that can learn probability distributions over its input data. It explains that RBMs define an energy function over hidden and visible units, with no connections between units within the same group. This conditional independence allows efficient computation of conditional probabilities. RBMs are trained using maximum likelihood, minimizing the negative log-likelihood of the training data by gradient descent.
Deepcoder to Self-Code with Machine Learning
Deepcoder to Self-Code with Machine LearningIRJET Journal The document discusses DeepCoder, a machine learning system developed by Microsoft that is able to generate its own code by learning from existing code examples. DeepCoder is trained on a large corpus of programs and input/output examples to learn which code snippets are likely to work together to solve new problems. It can then search through code more efficiently than humans to assemble working programs from existing code blocks. While currently limited to simple 5 line programs, DeepCoder represents a significant improvement over previous program synthesis techniques and could eventually make programming accessible to non-coders. However, some media reports exaggerated DeepCoder's capabilities and inaccurately claimed it works by copying code directly from other software.
Deep learning simplified
Deep learning simplifiedLovelyn Rose The document discusses deep learning and artificial neural networks. It provides an agenda for topics covered, including gradient descent, backpropagation, activation functions, and examples of neural network architectures like convolutional neural networks. It explains concepts like how neural networks learn patterns from data using techniques like stochastic gradient descent to minimize loss functions. Deep learning requires large amounts of processing power and labeled training data. Common deep learning networks are used for tasks like image recognition, object detection, and time series analysis.
Intelligent Parallel Processing and Compound Image Compression
Intelligent Parallel Processing and Compound Image CompressionDR.P.S.JAGADEESH KUMAR This document discusses parallel processing and compound image compression techniques. It examines the computational complexity and quantitative optimization of various image compression algorithms like BTC, DCT, DWT, DTCWT, SPIHT and EZW. The performance is evaluated in terms of coding efficiency, memory usage, image quality and quantity. Block Truncation Coding and Discrete Cosine Transform compression methods are described in more detail.
Towards Set Learning and Prediction - Laura Leal-Taixe - UPC Barcelona 2018
Towards Set Learning and Prediction - Laura Leal-Taixe - UPC Barcelona 2018Universitat Politècnica de Catalunya
Deep Generative Models I (DLAI D9L2 2017 UPC Deep Learning for Artificial Int...
Deep Generative Models I (DLAI D9L2 2017 UPC Deep Learning for Artificial Int...Universitat Politècnica de Catalunya
More from MLAI2 (20)
Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Unce...
Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Unce...MLAI2 Numerous recent works utilize bi-Lipschitz regularization of neural network layers to preserve relative distances between data instances in the feature spaces of each layer. This distance sensitivity with respect to the data aids in tasks such as uncertainty calibration and out-of-distribution (OOD) detection. In previous works, features extracted with a distance sensitive model are used to construct feature covariance matrices which are used in deterministic uncertainty estimation or OOD detection. However, in cases where there is a distribution over tasks, these methods result in covariances which are sub-optimal, as they may not leverage all of the meta information which can be shared among tasks. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. Additionally, we propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution which is well calibrated under a distributional dataset shift.
Online Hyperparameter Meta-Learning with Hypergradient Distillation
Online Hyperparameter Meta-Learning with Hypergradient DistillationMLAI2 Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on two different meta-learning methods and three benchmark datasets.
Online Coreset Selection for Rehearsal-based Continual Learning
Online Coreset Selection for Rehearsal-based Continual LearningMLAI2 A dataset is a shred of crucial evidence to describe a task. However, each data point in the dataset does not have the same potential, as some of the data points can be more representative or informative than others. This unequal importance among the data points may have a large impact in rehearsal-based continual learning, where we store a subset of the training examples (coreset) to be replayed later to alleviate catastrophic forgetting. In continual learning, the quality of the samples stored in the coreset directly affects the model's effectiveness and efficiency. The coreset selection problem becomes even more important under realistic settings, such as imbalanced continual learning or noisy data scenarios. To tackle this problem, we propose Online Coreset Selection (OCS), a simple yet effective method that selects the most representative and informative coreset at each iteration and trains them in an online manner. Our proposed method maximizes the model's adaptation to a target dataset while selecting high-affinity samples to past tasks, which directly inhibits catastrophic forgetting. We validate the effectiveness of our coreset selection mechanism over various standard, imbalanced, and noisy datasets against strong continual learning baselines, demonstrating that it improves task adaptation and prevents catastrophic forgetting in a sample-efficient manner.
Representational Continuity for Unsupervised Continual Learning
Representational Continuity for Unsupervised Continual LearningMLAI2 Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (Lump), a simple yet effective technique that interpolates between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations.
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual Learning
Sequential Reptile_Inter-Task Gradient Alignment for Multilingual LearningMLAI2 Multilingual models jointly pretrained on multiple languages have achieved remarkable performance on various multilingual downstream tasks. Moreover, models finetuned on a single monolingual downstream task have shown to generalize to unseen languages. In this paper, we first show that it is crucial for those tasks to align gradients between them in order to maximize knowledge transfer while minimizing negative transfer. Despite its importance, the existing methods for gradient alignment either have a completely different purpose, ignore inter-task alignment, or aim to solve continual learning problems in rather inefficient ways. As a result of the misaligned gradients between tasks, the model suffers from severe negative transfer in the form of catastrophic forgetting of the knowledge acquired from the pretraining. To overcome the limitations, we propose a simple yet effective method that can efficiently align gradients between tasks. Specifically, we perform each inner-optimization by sequentially sampling batches from all the tasks, followed by a Reptile outer update. Thanks to the gradients aligned between tasks by our method, the model becomes less vulnerable to negative transfer and catastrophic forgetting. We extensively validate our method on various multi-task learning and zero-shot cross-lingual transfer tasks, where our method largely outperforms all the relevant baselines we consider.
Skill-Based Meta-Reinforcement Learning
Skill-Based Meta-Reinforcement LearningMLAI2 While deep reinforcement learning methods have shown impressive results in robot learning, their sample inefficiency makes the learning of complex, long-horizon behaviors with real robot systems infeasible. To mitigate this issue, meta-reinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn. Yet, the application has been limited to short-horizon tasks with dense rewards. To enable learning long-horizon behaviors, recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations. While these approaches yield improved sample efficiency, millions of interactions with environments are still required to solve complex tasks. In this work, we devise a method that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning. Specifically, we propose to (1) extract reusable skills and a skill prior from offline datasets, (2) meta-train a high-level policy that learns to efficiently compose learned skills into long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve an unseen target task. Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve long-horizon novel target tasks by combining the strengths of meta-learning and the usage of offline datasets, while prior approaches in RL, meta-RL, and multi-task RL require substantially more environment interactions to solve the tasks.
Edge Representation Learning with Hypergraphs
Edge Representation Learning with HypergraphsMLAI2 Graph neural networks have recently achieved remarkable success in representing graph-structured data, with rapid progress in both the node embedding and graph pooling methods. Yet, they mostly focus on capturing information from the nodes considering their connectivity, and not much work has been done in representing the edges, which are essential components of a graph. However, for tasks such as graph reconstruction and generation, as well as graph classification tasks for which the edges are important for discrimination, accurately representing edges of a given graph is crucial to the success of the graph representation learning. To this end, we propose a novel edge representation learning framework based on Dual Hypergraph Transformation (DHT), which transforms the edges of a graph into the nodes of a hypergraph. This dual hypergraph construction allows us to apply message-passing techniques for node representations to edges. After obtaining edge representations from the hypergraphs, we then cluster or drop edges to obtain holistic graph-level edge representations. We validate our edge representation learning method with hypergraphs on diverse graph datasets for graph representation and generation performance, on which our method largely outperforms existing graph representation learning methods. Moreover, our edge representation learning and pooling method also largely outperforms state-of-theart graph pooling methods on graph classification, not only because of its accurate edge representation learning, but also due to its lossless compression of the nodes and removal of irrelevant edges for effective message-passing. Code is available at https://github.com/harryjo97/EHGNN.
Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Genera...
Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Genera...MLAI2 1) The document presents a method called FREED that uses fragment-based molecule generation guided by reinforcement learning to discover novel drug hits.
2) FREED explicitly constrains molecule generation to pharmacologically acceptable fragments to avoid toxic structures, which is more effective than implicit constraint methods.
3) FREED's exploratory RL algorithm prioritizes experience replay to encourage visiting novel states and finding diverse optima in the constrained chemical space.
Task Adaptive Neural Network Search with Meta-Contrastive Learning
Task Adaptive Neural Network Search with Meta-Contrastive LearningMLAI2 Most conventional Neural Architecture Search (NAS) approaches are limited in that they only generate architectures without searching for the optimal parameters. While some NAS methods handle this issue by utilizing a supernet trained on a large-scale dataset such as ImageNet, they may be suboptimal if the target tasks are highly dissimilar from the dataset the supernet is trained on. To address such limitations, we introduce a novel problem of Neural Network Search (NNS), whose goal is to search for the optimal pretrained network for a novel dataset and constraints (e.g. number of parameters), from a model zoo. Then, we propose a novel framework to tackle the problem, namely Task-Adaptive Neural Network Search (TANS). Given a model-zoo that consists of network pretrained on diverse datasets, we use a novel amortized meta-learning framework to learn a cross-modal latent space with contrastive loss, to maximize the similarity between a dataset and a high-performing network on it, and minimize the similarity between irrelevant dataset-network pairs. We validate the effectiveness and efficiency of our method on ten real-world datasets, against existing NAS/AutoML baselines. The results show that our method instantly retrieves networks that outperform models obtained with the baselines with significantly fewer training steps to reach the target performance, thus minimizing the total cost of obtaining a task-optimal network. Our code and the model-zoo are available at https://anonymous.4open.science/r/TANS-33D6
Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint L...
Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint L...MLAI2 While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning (FSSL). In this work, we study two essential scenarios of FSSL based on the location of the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server). We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning.
Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-Learning
Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-LearningMLAI2 Unsupervised learning aims to learn meaningful representations from unlabeled data which can captures its intrinsic structure, that can be transferred to downstream tasks. Meta-learning, whose objective is to learn to generalize across tasks such that the learned model can rapidly adapt to a novel task, shares the spirit of unsupervised learning in that the both seek to learn more effective and efficient learning procedure than learning from scratch. The fundamental difference of the two is that the most meta-learning approaches are supervised, assuming full access to the labels. However, acquiring labeled dataset for meta-training not only is costly as it requires human efforts in labeling but also limits its applications to pre-defined task distributions. In this paper, we propose a principled unsupervised meta-learning model, namely Meta-GMVAE, based on Variational Autoencoder (VAE) and set-level variational inference. Moreover, we introduce a mixture of Gaussian (GMM) prior, assuming that each modality represents each class-concept in a randomly sampled episode, which we optimize with Expectation-Maximization (EM). Then, the learned model can be used for downstream few-shot classification tasks, where we obtain task-specific parameters by performing semi-supervised EM on the latent representations of the support and query set, and predict labels of the query set by computing aggregated posteriors. We validate our model on Omniglot and Mini-ImageNet datasets by evaluating its performance on downstream few-shot classification tasks. The results show that our model obtain impressive performance gains over existing unsupervised meta-learning baselines, even outperforming supervised MAML on a certain setting.
Accurate Learning of Graph Representations with Graph Multiset Pooling
Accurate Learning of Graph Representations with Graph Multiset PoolingMLAI2 Graph neural networks have been widely used on modeling graph data, achieving impressive results on node classification and link prediction tasks. Yet, obtaining an accurate representation for a graph further requires a pooling function that maps a set of node representations into a compact form. A simple sum or average over all node representations considers all node features equally without consideration of their task relevance, and any structural dependencies among them. Recently proposed hierarchical graph pooling methods, on the other hand, may yield the same representation for two different graphs that are distinguished by the Weisfeiler-Lehman test, as they suboptimally preserve information from the node features. To tackle these limitations of existing graph pooling methods, we first formulate the graph pooling problem as a multiset encoding problem with auxiliary information about the graph structure, and propose a Graph Multiset Transformer (GMT) which is a multi-head attention based global pooling layer that captures the interaction between nodes according to their structural dependencies. We show that GMT satisfies both injectiveness and permutation invariance, such that it is at most as powerful as the Weisfeiler-Lehman graph isomorphism test. Moreover, our methods can be easily extended to the previous node clustering approaches for hierarchical graph pooling. Our experimental results show that GMT significantly outperforms state-of-the-art graph pooling methods on graph classification benchmarks with high memory and time efficiency, and obtains even larger performance gain on graph reconstruction and generation tasks.
Contrastive Learning with Adversarial Perturbations for Conditional Text Gene...
Contrastive Learning with Adversarial Perturbations for Conditional Text Gene...MLAI2 Recently, sequence-to-sequence (seq2seq) models with the Transformer architecture have achieved remarkable performance on various conditional text generation tasks, such as machine translation. However, most of them are trained with teacher forcing with the ground truth label given at each time step, without being exposed to incorrectly generated tokens during training, which hurts its generalization to unseen inputs, that is known as the “exposure bias” problem. In this work, we propose to mitigate the conditional text generation problem by contrasting positive pairs with negative pairs, such that the model is exposed to various valid or incorrect perturbations of the inputs, for improved generalization. However, training the model with naïve contrastive learning framework using random non-target sequences as negative examples is suboptimal, since they are easily distinguishable from the correct output, especially so with models pretrained with large text corpora. Also, generating positive examples requires domain-specific augmentation heuristics which may not generalize over diverse domains. To tackle this problem, we propose a principled method to generate positive and negative samples for contrastive learning of seq2seq models. Specifically, we generate negative examples by adding small perturbations to the input sequence to minimize its conditional likelihood, and positive examples by adding large perturbations while enforcing it to have a high conditional likelihood. Such “hard” positive and negative pairs generated using our method guides the model to better distinguish correct outputs from incorrect ones. We empirically show that our proposed method significantly improves the generalization of the seq2seq on three text generation tasks — machine translation, text summarization, and question generation.
Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Le...
Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Le...MLAI2 Although recent multi-task learning methods have shown to be effective in improving the generalization of deep neural networks, they should be used with caution for safety-critical applications, such as clinical risk prediction. This is because even if they achieve improved task-average performance, they may still yield degraded performance on individual tasks, which may be critical (e.g., prediction of mortality risk). Existing asymmetric multi-task learning methods tackle this negative transfer problem by performing knowledge transfer from tasks with low loss to tasks with high loss. However, using loss as a measure of reliability is risky since it could be a result of overfitting. In the case of time-series prediction tasks, knowledge learned for one task (e.g., predicting the sepsis onset) at a specific timestep may be useful for learning another task (e.g., prediction of mortality) at a later timestep, but lack of loss at each timestep makes it difficult to measure the reliability at each timestep. To capture such dynamically changing asymmetric relationships between tasks in time-series data, we propose a novel temporal asymmetric multi-task learning model that performs knowledge transfer from certain tasks/timesteps to relevant uncertain tasks, based on feature-level uncertainty. We validate our model on multiple clinical risk prediction tasks against various deep learning models for time-series prediction, which our model significantly outperforms, without any sign of negative transfer. Further qualitative analysis of learned knowledge graphs by clinicians shows that they are helpful in analyzing the predictions of the model. Our final code is available at this https://github.com/anhtuan5696/TPAMTL.
MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and Architectures
MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and ArchitecturesMLAI2 MetaPerturb is a meta-learned perturbation function that can enhance generalization of neural networks on different tasks and architectures. It proposes a novel meta-learning framework involving jointly training a main model and perturbation module on multiple source tasks to learn a transferable perturbation function. This meta-learned perturbation function can then be transferred to improve performance of a target model on an unseen target task or architecture, outperforming baselines on various datasets and architectures.
Adversarial Self-Supervised Contrastive Learning
Adversarial Self-Supervised Contrastive LearningMLAI2 Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions, which are then used to augment the training of the model for improved robustness. While some recent works propose semi-supervised adversarial learning methods that utilize unlabeled data, they still require class labels. However, do we really need class labels at all, for adversarially robust training of deep neural networks? In this paper, we propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. Further, we present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data, which aims to maximize the similarity between a random augmentation of a data sample and its instance-wise adversarial perturbation. We validate our method, Robust Contrastive Learning (RoCL), on multiple benchmark datasets, on which it obtains comparable robust accuracy over state-of-the-art supervised adversarial learning methods, and significantly improved robustness against the black box and unseen types of attacks. Moreover, with further joint fine-tuning with supervised adversarial loss, RoCL obtains even higher robust accuracy over using self-supervised learning alone. Notably, RoCL also demonstrate impressive results in robust transfer learning.
Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Pr...
Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Pr...MLAI2 Many practical graph problems, such as knowledge graph construction and drug-drug interaction prediction, require to handle multi-relational graphs. However, handling real-world multi-relational graphs with Graph Neural Networks (GNNs) is often challenging due to their evolving nature, as new entities (nodes) can emerge over time. Moreover, newly emerged entities often have few links, which makes the learning even more difficult. Motivated by this challenge, we introduce a realistic problem of few-shot out-of-graph link prediction, where we not only predict the links between the seen and unseen nodes as in a conventional out-of-knowledge link prediction task but also between the unseen nodes, with only few edges per node. We tackle this problem with a novel transductive meta-learning framework which we refer to as Graph Extrapolation Networks (GEN). GEN meta-learns both the node embedding network for inductive inference (seen-to-unseen) and the link prediction network for transductive inference (unseen-to-unseen). For transductive link prediction, we further propose a stochastic embedding layer to model uncertainty in the link prediction between unseen entities. We validate our model on multiple benchmark datasets for knowledge graph completion and drug-drug interaction prediction. The results show that our model significantly outperforms relevant baselines for out-of-graph link prediction tasks.
Neural Mask Generator : Learning to Generate Adaptive Word Maskings for Langu...
Neural Mask Generator : Learning to Generate Adaptive Word Maskings for Langu...MLAI2 We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.
Cost-effective Interactive Attention Learning with Neural Attention Process
Cost-effective Interactive Attention Learning with Neural Attention ProcessMLAI2 We propose a novel interactive learning framework which we refer to as Interactive Attention Learning (IAL), in which the human supervisors interactively manipulate the allocated attentions, to correct the model's behavior by updating the attention-generating network. However, such a model is prone to overfitting due to scarcity of human annotations, and requires costly retraining. Moreover, it is almost infeasible for the human annotators to examine attentions on tons of instances and features. We tackle these challenges by proposing a sample-efficient attention mechanism and a cost-effective reranking algorithm for instances and features. First, we propose Neural Attention Process (NAP), which is an attention generator that can update its behavior by incorporating new attention-level supervisions without any retraining. Secondly, we propose an algorithm which prioritizes the instances and the features by their negative impacts, such that the model can yield large improvements with minimal human feedback. We validate IAL on various time-series datasets from multiple domains (healthcare, real-estate, and computer vision) on which it significantly outperforms baselines with conventional attention mechanisms, or without cost-effective reranking, with substantially less retraining and human-model interaction cost.
Adversarial Neural Pruning with Latent Vulnerability Suppression
Adversarial Neural Pruning with Latent Vulnerability SuppressionMLAI2 Despite the remarkable performance of deep neural networks on various computer vision tasks, they are known to be susceptible to adversarial perturbations, which makes it challenging to deploy them in real-world safety-critical applications. In this paper, we conjecture that the leading cause of adversarial vulnerability is the distortion in the latent feature space, and provide methods to suppress them effectively. Explicitly, we define vulnerability for each latent feature and then propose a new loss for adversarial learning, Vulnerability Suppression (VS) loss, that aims to minimize the feature-level vulnerability during training. We further propose a Bayesian framework to prune features with high vulnerability to reduce both vulnerability and loss on adversarial samples. We validate our Adversarial Neural Pruning with Vulnerability Suppression (ANP-VS) method on multiple benchmark datasets, on which it not only obtains state-of-the-art adversarial robustness but also improves the performance on clean examples, using only a fraction of the parameters used by the full network. Further qualitative analysis suggests that the improvements come from the suppression of feature-level vulnerability.
Ad
Recently uploaded (20)
Nix(OS) for Python Developers - PyCon 25 (Bologna, Italia)
Nix(OS) for Python Developers - PyCon 25 (Bologna, Italia)Peter Bittner How do you onboard new colleagues in 2025? How long does it take? Would you love a standardized setup under version control that everyone can customize for themselves? A stable desktop setup, reinstalled in just minutes. It can be done.
This talk was given in Italian, 29 May 2025, at PyCon 25, Bologna, Italy. All slides are provided in English.
Original slides at https://slides.com/bittner/pycon25-nixos-for-python-developers
Introducing the OSA 3200 SP and OSA 3250 ePRC
Introducing the OSA 3200 SP and OSA 3250 ePRCAdtran Adtran's latest Oscilloquartz solutions make optical pumping cesium timing more accessible than ever. Discover how the new OSA 3200 SP and OSA 3250 ePRC deliver superior stability, simplified deployment and lower total cost of ownership. Built on a shared platform and engineered for scalable, future-ready networks, these models are ideal for telecom, defense, metrology and more.
TrustArc Webinar: Mastering Privacy Contracting
TrustArc Webinar: Mastering Privacy ContractingTrustArc As data privacy regulations become more pervasive across the globe and organizations increasingly handle and transfer (including across borders) meaningful volumes of personal and confidential information, the need for robust contracts to be in place is more important than ever.
This webinar will provide a deep dive into privacy contracting, covering essential terms and concepts, negotiation strategies, and key practices for managing data privacy risks.
Whether you're in legal, privacy, security, compliance, GRC, procurement, or otherwise, this session will include actionable insights and practical strategies to help you enhance your agreements, reduce risk, and enable your business to move fast while protecting itself.
This webinar will review key aspects and considerations in privacy contracting, including:
- Data processing addenda, cross-border transfer terms including EU Model Clauses/Standard Contractual Clauses, etc.
- Certain legally-required provisions (as well as how to ensure compliance with those provisions)
- Negotiation tactics and common issues
- Recent lessons from recent regulatory actions and disputes
UiPath Community Berlin: Studio Tips & Tricks and UiPath Insights
UiPath Community Berlin: Studio Tips & Tricks and UiPath InsightsUiPathCommunity Join the UiPath Community Berlin (Virtual) meetup on May 27 to discover handy Studio Tips & Tricks and get introduced to UiPath Insights. Learn how to boost your development workflow, improve efficiency, and gain visibility into your automation performance.
📕 Agenda:
- Welcome & Introductions
- UiPath Studio Tips & Tricks for Efficient Development
- Best Practices for Workflow Design
- Introduction to UiPath Insights
- Creating Dashboards & Tracking KPIs (Demo)
- Q&A and Open Discussion
Perfect for developers, analysts, and automation enthusiasts!
This session streamed live on May 27, 18:00 CET.
Check out all our upcoming UiPath Community sessions at:
👉 https://community.uipath.com/events/
Join our UiPath Community Berlin chapter:
👉 https://community.uipath.com/berlin/
Cybersecurity Fundamentals: Apprentice - Palo Alto Certificate
Cybersecurity Fundamentals: Apprentice - Palo Alto CertificateVICTOR MAESTRE RAMIREZ Cybersecurity Fundamentals: Apprentice - Palo Alto Certificate
Securiport - A Border Security Company
Securiport - A Border Security CompanySecuriport Securiport is a border security systems provider with a progressive team approach to its task. The company acknowledges the importance of specialized skills in creating the latest in innovative security tech. The company has offices throughout the world to serve clients, and its employees speak more than twenty languages at the Washington D.C. headquarters alone.
European Accessibility Act & Integrated Accessibility Testing
European Accessibility Act & Integrated Accessibility TestingJulia Undeutsch Emma Dawson will guide you through two important topics in this session.
Firstly, she will prepare you for the European Accessibility Act (EAA), which comes into effect on 28 June 2025, and show you how development teams can prepare for it.
In the second part of the webinar, Emma Dawson will explore with you various integrated testing methods and tools that will help you improve accessibility during the development cycle, such as Linters, Storybook, Playwright, just to name a few.
Focus: European Accessibility Act, Integrated Testing tools and methods (e.g. Linters, Storybook, Playwright)
Target audience: Everyone, Developers, Testers
Droidal: AI Agents Revolutionizing Healthcare
Droidal: AI Agents Revolutionizing HealthcareDroidal LLC Droidal’s AI Agents are transforming healthcare by bringing intelligence, speed, and efficiency to key areas such as Revenue Cycle Management (RCM), clinical operations, and patient engagement. Built specifically for the needs of U.S. hospitals and clinics, Droidal's solutions are designed to improve outcomes and reduce administrative burden.
Through simple visuals and clear examples, the presentation explains how AI Agents can support medical coding, streamline claims processing, manage denials, ensure compliance, and enhance communication between providers and patients. By integrating seamlessly with existing systems, these agents act as digital coworkers that deliver faster reimbursements, reduce errors, and enable teams to focus more on patient care.
Droidal's AI technology is more than just automation — it's a shift toward intelligent healthcare operations that are scalable, secure, and cost-effective. The presentation also offers insights into future developments in AI-driven healthcare, including how continuous learning and agent autonomy will redefine daily workflows.
Whether you're a healthcare administrator, a tech leader, or a provider looking for smarter solutions, this presentation offers a compelling overview of how Droidal’s AI Agents can help your organization achieve operational excellence and better patient outcomes.
A free demo trial is available for those interested in experiencing Droidal’s AI Agents firsthand. Our team will walk you through a live demo tailored to your specific workflows, helping you understand the immediate value and long-term impact of adopting AI in your healthcare environment.
To request a free trial or learn more:
https://droidal.com/
ECS25 - The adventures of a Microsoft 365 Platform Owner - Website.pptx
ECS25 - The adventures of a Microsoft 365 Platform Owner - Website.pptxJasper Oosterveld My slides for ECS 2025.
SDG 9000 Series: Unleashing multigigabit everywhere
SDG 9000 Series: Unleashing multigigabit everywhereAdtran Adtran’s SDG 9000 Series brings high-performance, cloud-managed Wi-Fi 7 to homes, businesses and public spaces. Built on a unified SmartOS platform, the portfolio includes outdoor access points, ceiling-mount APs and a 10G PoE router. Intellifi and Mosaic One simplify deployment, deliver AI-driven insights and unlock powerful new revenue streams for service providers.
Dev Dives: System-to-system integration with UiPath API Workflows
Dev Dives: System-to-system integration with UiPath API WorkflowsUiPathCommunity Join the next Dev Dives webinar on May 29 for a first contact with UiPath API Workflows, a powerful tool purpose-fit for API integration and data manipulation!
This session will guide you through the technical aspects of automating communication between applications, systems and data sources using API workflows.
📕 We'll delve into:
- How this feature delivers API integration as a first-party concept of the UiPath Platform.
- How to design, implement, and debug API workflows to integrate with your existing systems seamlessly and securely.
- How to optimize your API integrations with runtime built for speed and scalability.
This session is ideal for developers looking to solve API integration use cases with the power of the UiPath Platform.
👨🏫 Speakers:
Gunter De Souter, Sr. Director, Product Manager @UiPath
Ramsay Grove, Product Manager @UiPath
This session streamed live on May 29, 2025, 16:00 CET.
Check out all our upcoming UiPath Dev Dives sessions:
👉 https://community.uipath.com/dev-dives-automation-developer-2025/
New Ways to Reduce Database Costs with ScyllaDB
New Ways to Reduce Database Costs with ScyllaDBScyllaDB How ScyllaDB’s latest capabilities can reduce your infrastructure costs
ScyllaDB has been obsessed with price-performance from day 1. Our core database is architected with low-level engineering optimizations that squeeze every ounce of power from the underlying infrastructure. And we just completed a multi-year effort to introduce a set of new capabilities for additional savings.
Join this webinar to learn about these new capabilities: the underlying challenges we wanted to address, the workloads that will benefit most from each, and how to get started. We’ll cover ways to:
- Avoid overprovisioning with “just-in-time” scaling
- Safely operate at up to ~90% storage utilization
- Cut network costs with new compression strategies and file-based streaming
We’ll also highlight a “hidden gem” capability that lets you safely balance multiple workloads in a single cluster. To conclude, we will share the efficiency-focused capabilities on our short-term and long-term roadmaps.
Microsoft Build 2025 takeaways in one presentation
Microsoft Build 2025 takeaways in one presentationDigitalmara Microsoft Build 2025 introduced significant updates. Everything revolves around AI. DigitalMara analyzed these announcements:
• AI enhancements for Windows 11
By embedding AI capabilities directly into the OS, Microsoft is lowering the barrier for users to benefit from intelligent automation without requiring third-party tools. It's a practical step toward improving user experience, such as streamlining workflows and enhancing productivity. However, attention should be paid to data privacy, user control, and transparency of AI behavior. The implementation policy should be clear and ethical.
• GitHub Copilot coding agent
The introduction of coding agents is a meaningful step in everyday AI assistance. However, it still brings challenges. Some people compare agents with junior developers. They noted that while the agent can handle certain tasks, it often requires supervision and can introduce new issues. This innovation holds both potential and limitations. Balancing automation with human oversight is crucial to ensure quality and reliability.
• Introduction of Natural Language Web
NLWeb is a significant step toward a more natural and intuitive web experience. It can help users access content more easily and reduce reliance on traditional navigation. The open-source foundation provides developers with the flexibility to implement AI-driven interactions without rebuilding their existing platforms. NLWeb is a promising level of web interaction that complements, rather than replaces, well-designed UI.
• Introduction of Model Context Protocol
MCP provides a standardized method for connecting AI models with diverse tools and data sources. This approach simplifies the development of AI-driven applications, enhancing efficiency and scalability. Its open-source nature encourages broader adoption and collaboration within the developer community. Nevertheless, MCP can face challenges in compatibility across vendors and security in context sharing. Clear guidelines are crucial.
• Windows Subsystem for Linux is open-sourced
It's a positive step toward greater transparency and collaboration in the developer ecosystem. The community can now contribute to its evolution, helping identify issues and expand functionality faster. However, open-source software in a core system also introduces concerns around security, code quality management, and long-term maintenance. Microsoft’s continued involvement will be key to ensuring WSL remains stable and secure.
• Azure AI Foundry platform hosts Grok 3 AI models
Adding new models is a valuable expansion of AI development resources available at Azure. This provides developers with more flexibility in choosing language models that suit a range of application sizes and needs. Hosting on Azure makes access and integration easier when using Microsoft infrastructure.
Agentic AI - The New Era of Intelligence
Agentic AI - The New Era of IntelligenceMuzammil Shah This presentation is specifically designed to introduce final-year university students to the foundational principles of Agentic Artificial Intelligence (AI). It aims to provide a clear understanding of how Agentic AI systems function, their key components, and the underlying technologies that empower them. By exploring real-world applications and emerging trends, the session will equip students with essential knowledge to engage with this rapidly evolving area of AI, preparing them for further study or professional work in the field.
Maxx nft market place new generation nft marketing place
Maxx nft market place new generation nft marketing placeusersalmanrazdelhi PREFACE OF MAXXNFT
MaxxNFT: Powering the Future of Digital Ownership
MaxxNFT is a cutting-edge Web3 platform designed to revolutionize how
digital assets are owned, traded, and valued. Positioned at the forefront of the
NFT movement, MaxxNFT views NFTs not just as collectibles, but as the next
generation of internet equity—unique, verifiable digital assets that unlock new
possibilities for creators, investors, and everyday users alike.
Through strategic integrations with OKT Chain and OKX Web3, MaxxNFT
enables seamless cross-chain NFT trading, improved liquidity, and enhanced
user accessibility. These collaborations make it easier than ever to participate
in the NFT ecosystem while expanding the platform’s global reach.
With a focus on innovation, user rewards, and inclusive financial growth,
MaxxNFT offers multiple income streams—from referral bonuses to liquidity
incentives—creating a vibrant community-driven economy. Whether you
'
re
minting your first NFT or building a digital asset portfolio, MaxxNFT empowers
you to participate in the future of decentralized value exchange.
https://maxxnft.xyz/
Grannie’s Journey to Using Healthcare AI Experiences
Grannie’s Journey to Using Healthcare AI ExperiencesLauren Parr AI offers transformative potential to enhance our long-time persona Grannie’s life, from healthcare to social connection. This session explores how UX designers can address unmet needs through AI-driven solutions, ensuring intuitive interfaces that improve safety, well-being, and meaningful interactions without overwhelming users.
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025Lorenzo Miniero Slides for my "Multistream support in the Janus SIP and NoSIP plugins" presentation at the OpenSIPS Summit 2025 event.
They describe my efforts refactoring the Janus SIP and NoSIP plugins to allow for the gatewaying of an arbitrary number of audio/video streams per call (thus breaking the current 1-audio/1-video limitation), plus some additional considerations on what this could mean when dealing with application protocols negotiated via SIP as well.
Introducing FME Realize: A New Era of Spatial Computing and AR
Introducing FME Realize: A New Era of Spatial Computing and ARSafe Software A new era for the FME Platform has arrived – and it’s taking data into the real world.
Meet FME Realize: marking a new chapter in how organizations connect digital information with the physical environment around them. With the addition of FME Realize, FME has evolved into an All-data, Any-AI Spatial Computing Platform.
FME Realize brings spatial computing, augmented reality (AR), and the full power of FME to mobile teams: making it easy to visualize, interact with, and update data right in the field. From infrastructure management to asset inspections, you can put any data into real-world context, instantly.
Join us to discover how spatial computing, powered by FME, enables digital twins, AI-driven insights, and real-time field interactions: all through an intuitive no-code experience.
In this one-hour webinar, you’ll:
-Explore what FME Realize includes and how it fits into the FME Platform
-Learn how to deliver real-time AR experiences, fast
-See how FME enables live, contextual interactions with enterprise data across systems
-See demos, including ones you can try yourself
-Get tutorials and downloadable resources to help you start right away
Whether you’re exploring spatial computing for the first time or looking to scale AR across your organization, this session will give you the tools and insights to get started with confidence.
UiPath Community Zurich: Release Management and Build Pipelines
UiPath Community Zurich: Release Management and Build PipelinesUiPathCommunity Ensuring robust, reliable, and repeatable delivery processes is more critical than ever - it's a success factor for your automations and for automation programmes as a whole. In this session, we’ll dive into modern best practices for release management and explore how tools like the UiPathCLI can streamline your CI/CD pipelines. Whether you’re just starting with automation or scaling enterprise-grade deployments, our event promises to deliver helpful insights to you. This topic is relevant for both on-premise and cloud users - as well as for automation developers and software testers alike.
📕 Agenda:
- Best Practices for Release Management
- What it is and why it matters
- UiPath Build Pipelines Deep Dive
- Exploring CI/CD workflows, the UiPathCLI and showcasing scenarios for both on-premise and cloud
- Discussion, Q&A
👨🏫 Speakers
Roman Tobler, CEO@ Routinuum
Johans Brink, CTO@ MvR Digital Workforce
We look forward to bringing best practices and showcasing build pipelines to you - and to having interesting discussions on this important topic!
If you have any questions or inputs prior to the event, don't hesitate to reach out to us.
This event streamed live on May 27, 16:00 pm CET.
Check out all our upcoming UiPath Community sessions at:
👉 https://community.uipath.com/events/
Join UiPath Community Zurich chapter:
👉 https://community.uipath.com/zurich/
AI Trends - Mary Meeker
AI Trends - Mary MeekerRazin Mustafiz Source: https://www.bondcap.com/reports/tai
Ad
Mini-Batch Consistent Slot Set Encoder For Scalable Set Encoding
- 1. Mini-Batch Consistent Slot Set Encoder For Scalable Set Encoding Andreis Bruno1, Jeffrey Ryan Willette1, Juho Lee1,2, Sung Ju Hwang1,2 1KAIST, South Korea 2AITRICS, South Korea 1
- 2. Many problems in machine learning involve converting a set of arbitrary size to a single vector or set of vectors, the set encoding/representation. The Set Encoding Problem Encoder Set Encoding This places a few symmetrical (sometimes probabilistic) restrictions on the encoder. 2
- 3. Property 1 A function 𝒇: 𝟐𝑿 → 𝒀 acting on sets must be permutation invariant to the order of objects in the set, i.e. for any permutation 𝝅: 𝒇 𝒙𝟏, … , 𝒙𝑴 = 𝒇 𝒙𝝅 𝟏 , … , 𝒙𝝅 𝑴 . Exchangeability A distribution for a set of random variables 𝑿 = 𝒙𝒊 𝒊&𝟏 𝑴 is exchangeable if for an permutation 𝝅: 𝒑 𝑿 = 𝒑 𝝅 𝑿 . Property 2 A function 𝒇: 𝑿𝑴 → 𝒀𝑴 acting on sets is a permutation equivariant function if permutation of the input instances permutes the output labels, i.e. for any permutation 𝝅: 𝒇 𝒙𝝅 𝟏 , … , 𝒙𝝅 𝑴 = [𝒇𝝅 𝟏 𝒙 , … , 𝒇𝝅 𝑴 𝒙 ] Permutation Invariance & Equivariance Bloem-Reddy, Benjamin, and Yee Whye Teh. "Probabilistic Symmetries and Invariant Neural Networks." J. Mach. Learn. Res. 21 (2020): 90-1. 3
- 4. Mini-Batch Consistent (MBC) Set Encoding Given large sets, we want to be able to process the elements of the set in mini- batches based on the available computational and memory resources. Set encoders such as DeepSets and Set Transformers can be modified to do this but not all can perform mini-batch encoding consistently. We formalize the requirements for MBC set encoding below: Property 5 𝐿𝑒𝑡 𝑿 ∈ 𝑹𝑴×𝒅 𝑏𝑒 𝑝𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛𝑒𝑑 𝑠𝑢𝑐ℎ 𝑡ℎ𝑎𝑡 𝑿 = 𝑿𝟏 ∪ 𝑿𝟐 ∪ ⋯ ∪ 𝑿𝒑 𝑎𝑛𝑑 𝒇: 𝑹𝑴𝒊×𝒅 → 𝑹𝒅" 𝑏𝑒 𝑎 𝑠𝑒𝑡 𝑒𝑛𝑐𝑜𝑑𝑖𝑛𝑔 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑠𝑢𝑐ℎ 𝑡ℎ𝑎𝑡 𝒇 𝑿 = 𝒁. 𝐺𝑖𝑣𝑒𝑛 𝑎𝑛 𝑎𝑔𝑔𝑟𝑒𝑔𝑎𝑡𝑖𝑜𝑛 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝒈: 𝒁𝒋 ∈ 𝑹𝒅" 𝒋&𝟏 𝒑 → 𝑹𝒅" , 𝒈 𝑎𝑛𝑑 𝒇 𝑎𝑟𝑒 𝑀𝑖𝑛𝑖 − 𝐵𝑎𝑡𝑐ℎ 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑡 𝑖𝑓 𝑎𝑛𝑑 𝑜𝑛𝑙𝑦 𝑖𝑓 𝒈 𝒇 𝑿𝟏 , … , 𝒇 𝑿𝒑 = 𝒇 𝑿 . Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 4
- 5. Violation of MBC: Set Transformer We train Set Transformer on an image reconstruction task. At test time, we increase the number of pixels and encode them in a mini-batch fashion. The performance of the model degrades in the mini-batch setting. Additionally, it is not immediately clear how to aggregate the encodings of the mini-batches. Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 5
- 6. MBC Set Encoding Deep Sets can trivially satisfy MBC by removing the message-passing layers. Set Transformer, which is attention based, violates MBC. Our goal is to design an attention based set encoder, such as Set Transformer, that satisfies MBC. We achieve this by using slots. Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 6
- 7. Slot Set Encoder (SSE) We realize an MBC set encoder, SSE, by computing attention over slots instead of between the elements of the set. This makes SSE amenable to mini-batch processing. The SSE in Algorithm 1 is functionally composable over partitions of the input X for a given slot initialization and any partition of X. 𝑺 ∼ 𝑁 𝜇, 𝑑𝑖𝑎𝑔 𝜎 ∈ 𝑅!×# 𝒂𝒕𝒕𝒏𝒊,𝒋 ≔ 𝝈 𝑀',( 𝑤ℎ𝑒𝑟𝑒 𝑀 ≔ 1 C 𝑑 𝑘 𝑿 ⋅ 𝑞 𝑺 ) ∈ 𝑅*×+ G 𝑺 ≔ 𝑾𝑻 ⋅ 𝑣 𝑿 ∈ 𝑅!× - . 𝑤ℎ𝑒𝑟𝑒 𝑾𝒊,𝒋 ≔ 𝒂𝒕𝒕𝒏𝒊,𝒋 ∑/01 ! 𝒂𝒕𝒕𝒏𝒊,𝒍 𝒇 𝑿 = 𝒈 𝒇 𝑿𝟏 , 𝒇 𝑿𝟐 , … , 𝒇 𝑿𝒑 𝒈 ∈ {𝑚𝑒𝑎𝑛, 𝑠𝑢𝑚, 𝑚𝑎𝑥, 𝑚𝑖𝑛) Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 7
- 8. Slot Set Encoder (SSE) SSE is permutation invariant with respect to partitions of the input set in permutation equivariant with respect to the order of slots. Proposition 3 𝐹𝑜𝑟 𝑎 𝑔𝑖𝑣𝑒𝑛 𝑖𝑛𝑝𝑢𝑡 𝑠𝑒𝑡 𝑋 ∈ 𝑅V×W 𝑎𝑛𝑑 𝑠𝑙𝑜𝑡 𝑖𝑛𝑖𝑡𝑖𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛𝑠 𝑆 ∈ 𝑅X×W, 𝑡ℎ𝑒 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑠 𝑓 𝑎𝑛𝑑 𝑔 𝑎𝑠 𝑑𝑒𝑓𝑖𝑛𝑒𝑑 𝑖𝑛 𝐴𝑙𝑔𝑜𝑟𝑖𝑡ℎ𝑚 1 𝑎𝑟𝑒 𝑀𝐵𝐶 𝑓𝑜𝑟 𝒂𝒏𝒚 𝒑𝒂𝒓𝒕𝒊𝒕𝒊𝒐𝒏 𝒐𝒇 𝑿 𝑎𝑛𝑑 ℎ𝑒𝑛𝑐𝑒 𝑠𝑎𝑡𝑖𝑠𝑓𝑦 𝑡ℎ𝑒 𝑀𝐵𝐶 𝑝𝑟𝑜𝑝𝑒𝑟𝑡𝑦. Proposition 4 𝐿𝑒𝑡 𝑋 ∈ 𝑅V×W 𝑎𝑛𝑑 𝑆 ∈ 𝑅X×W 𝑏𝑒 𝑎𝑛 𝑖𝑛𝑝𝑢𝑡 𝑠𝑒𝑡 𝑎𝑛𝑑 𝑠𝑙𝑜𝑡 𝑖𝑛𝑖𝑡𝑖𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛 𝑟𝑒𝑠𝑝𝑒𝑐𝑡𝑖𝑣𝑒𝑙𝑦. 𝐴𝑑𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙𝑙𝑦, 𝑙𝑒𝑡 𝑆𝑆𝐸 𝑋, 𝑆 𝑏𝑒 𝑡ℎ𝑒 𝑜𝑢𝑡𝑝𝑢𝑡 𝑜𝑓 𝐴𝑙𝑔𝑜𝑟𝑖𝑡ℎ𝑚 1, 𝑎𝑛𝑑 𝜋Y ∈ 𝑅V×V 𝑎𝑛𝑑 𝜋Z ∈ 𝑅X×X 𝑏𝑒 𝑎𝑟𝑏𝑖𝑡𝑟𝑎𝑟𝑦 𝑝𝑒𝑟𝑚𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝑚𝑎𝑡𝑟𝑖𝑐𝑒𝑠. 𝑇ℎ𝑒𝑛 𝑺𝑺𝑬 𝝅𝐱 ⋅ 𝑿, 𝝅𝑺 ⋅ 𝑺 = 𝝅𝑺 ⋅ 𝑺𝑺𝑬(𝑿, 𝑺) Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 8
- 9. Hierarchical Slot Set Encoder We can stack multiple Slot Set Encoders on top of each to obtain a hierarchy of slot set encoders. This allows us to model higher-order interactions across slots. 𝑓 𝑋 = 𝑆𝑆𝐸 … 𝑆𝑆𝐸] 𝑆𝑆𝐸^ 𝑋 The resulting set encoding function 𝑓(𝑋) satisfies the MBC property as well as Propositions 3 & 4. Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 9
- 10. Approximate Mini-Batch Training of MBC Encoders How can we train Slot Set Encoders in the large scale or streaming setting? Both DeepSets and Set Transformers require gradients to be taken with respect to the full set at train time. In the Mini-Batch Consistent Setting, this is not feasible for large sets or when set elements arrive in a stream. We train MBC models on partitions of sets sampled at each iteration of the optimization process and find that it works well empirically. Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 10
- 11. Experiments: Point Cloud Classification (ModelNet40) We first show that SSE is a valid set encoding function on the point cloud classification task. Here, no mini-batch encoding is used. Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. Encoder Set Encoding Classifier 11
- 12. Experiments: Image Reconstruction (CelebA) We perform image reconstruction using Conditional Neural Processes. We replace the aggregation function with DeepSets, Set Transformer or Slot Set Encoder. We test this model in the mini-batch setting where data arrives in a stream. Andreis, B., Willette, J., Lee, J., & Hwang, S. J. (2021). Mini-Batch Consistent Slot Set Encoder for Scalable Set Encoding. arXiv preprint arXiv:2103.01615. 12