Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Prediction

Dec 26, 20200 likes977 views

Many practical graph problems, such as knowledge graph construction and drug-drug interaction prediction, require to handle multi-relational graphs. However, handling real-world multi-relational graphs with Graph Neural Networks (GNNs) is often challenging due to their evolving nature, as new entities (nodes) can emerge over time. Moreover, newly emerged entities often have few links, which makes the learning even more difficult. Motivated by this challenge, we introduce a realistic problem of few-shot out-of-graph link prediction, where we not only predict the links between the seen and unseen nodes as in a conventional out-of-knowledge link prediction task but also between the unseen nodes, with only few edges per node. We tackle this problem with a novel transductive meta-learning framework which we refer to as Graph Extrapolation Networks (GEN). GEN meta-learns both the node embedding network for inductive inference (seen-to-unseen) and the link prediction network for transductive inference (unseen-to-unseen). For transductive link prediction, we further propose a stochastic embedding layer to model uncertainty in the link prediction between unseen entities. We validate our model on multiple benchmark datasets for knowledge graph completion and drug-drug interaction prediction. The results show that our model significantly outperforms relevant baselines for out-of-graph link prediction tasks.

Learning to Extrapolate Knowledge:
Transductive Few-shot Out-of-Graph Link Prediction
Jinheon Baek1, Dong Bok Lee1, Sung Ju Hwang1,2
1Graduate School of AI, KAIST, South Korea
2AITRICS, South Korea

While graphs contain and express a huge amount of knowledge, they are highly
incomplete.
Link Prediction
(A) Incomplete knowledge graph. (B) Predicting missing links.
Thus automatic graph completion, known as link prediction, is practically important.

• Have an evolving nature, where new entities can emerge over time.
• Exhibit long-tail distributions, where most entities have few triplets to train.
Challenges on Real-world Graphs
(A) Evolving nature. (B) Long-tail distribution.

We propose a few-shot out-of-graph link prediction problem whose goal is to predict
links between seen and unseen, or among unseen entities, with few links per entity.
Meta-Learning Framework

To tackle the out-of-graph link prediction problem, we propose a novel meta-
learning framework, which meta-learns the node embedding for unseen entities.
Meta-Learning Framework
Our meta-learning framework learns by simulating the unseen entities during
training, and extrapolates this knowledge to the real unseen entities.
Training a network with massively
generated simulated unseen entities.

This meta-learning makes the model generalize well to the link prediction tasks on
unseen out-of-graph entities.
Meta-Learning Framework
Generalization over real unseen
entities with meta-learned network.

(Inductive) GEN learns to predict links between seen and unseen entities with
output embedding, by simulating unseen entities with seen entities.
Graph Extrapolation Network (GEN)
(A) Meta-learning framework. (B) Meta-learned Network
(Graph Extrapolation Network).

(Transductive) GEN further learns to predict the links even among unseen entities,
with simulated unseen entities during meta-training.
Graph Extrapolation Network (GEN)
(A) Meta-learning framework. (B) Meta-learned Network
(Graph Extrapolation Network).

Results
Transductive-GEN (T-GEN) outperforms all baselines on out-of-graph link prediction
tasks for knowledge graph completion and drug-drug interaction prediction.
FB15-237 NELL-995
Types Models MRR Hits@10 MRR Hits@10
Seen-to-Seen
TransE 0.053 0.082 0.009 0.020
R-GCN 0.008 0.011 0.004 0.007
Seen-to-Seen, re-
trained from scratch
TransE 0.071 0.159 0.071 0.129
R-GCN 0.099 0.181 0.112 0.184
Seen-to-Unseen
MEAN 0.105 0.207 0.158 0.263
LAN 0.112 0.214 0.159 0.255
Ours T-GEN 0.367 0.530 0.282 0.421
(A) Knowledge Graph Completion.
DeepDDI BIOSNAP-sub
Types Models PR Acc PR Acc
Seen-to-Seen,
re-trained from
scratch
MLP 0.476 0.528 0.034 0.049
MPNN 0.478 0.681 0.026 0.067
R-GCN 0.397 0.640 0.041 0.051
Ours T-GEN 0.708 0.815 0.067 0.089
(B) Drug-Drug Interaction Prediction.

Results
Why does GEN generalize well to link prediction with out-of-graph entities?
This is because GEN embeds the unseen entities on the manifold of seen entities,
while baselines embeds the unseen entities off-manifold.
(A) Seen-to-Unseen Baseline
(LAN [Wang et al.]).
(B) Seen-to-Seen Baseline, retrained
from scratch (TransE [Bordes et al.]).
(C) Ours (T-GEN).

Conclusion
• We define a realistic problem setting of few-shot out-of-graph link prediction,
aiming to perform link prediction for unseen entities.
• To tackle this problem, we propose a novel meta-learning framework, which
meta-learns the node embedding for unseen entities.
• We validate our model on knowledge graph completion and drug-drug
interaction tasks, on which it significantly outperforms relevant baselines.

Graph neural networks have recently achieved remarkable success in representing graph-structured data, with rapid progress in both the node embedding and graph pooling methods. Yet, they mostly focus on capturing information from the nodes considering their connectivity, and not much work has been done in representing the edges, which are essential components of a graph. However, for tasks such as graph reconstruction and generation, as well as graph classification tasks for which the edges are important for discrimination, accurately representing edges of a given graph is crucial to the success of the graph representation learning. To this end, we propose a novel edge representation learning framework based on Dual Hypergraph Transformation (DHT), which transforms the edges of a graph into the nodes of a hypergraph. This dual hypergraph construction allows us to apply message-passing techniques for node representations to edges. After obtaining edge representations from the hypergraphs, we then cluster or drop edges to obtain holistic graph-level edge representations. We validate our edge representation learning method with hypergraphs on diverse graph datasets for graph representation and generation performance, on which our method largely outperforms existing graph representation learning methods. Moreover, our edge representation learning and pooling method also largely outperforms state-of-theart graph pooling methods on graph classification, not only because of its accurate edge representation learning, but also due to its lossless compression of the nodes and removal of irrelevant edges for effective message-passing. Code is available at https://github.com/harryjo97/EHGNN.

Accurate Learning of Graph Representations with Graph Multiset PoolingMLAI2

Graph neural networks have been widely used on modeling graph data, achieving impressive results on node classification and link prediction tasks. Yet, obtaining an accurate representation for a graph further requires a pooling function that maps a set of node representations into a compact form. A simple sum or average over all node representations considers all node features equally without consideration of their task relevance, and any structural dependencies among them. Recently proposed hierarchical graph pooling methods, on the other hand, may yield the same representation for two different graphs that are distinguished by the Weisfeiler-Lehman test, as they suboptimally preserve information from the node features. To tackle these limitations of existing graph pooling methods, we first formulate the graph pooling problem as a multiset encoding problem with auxiliary information about the graph structure, and propose a Graph Multiset Transformer (GMT) which is a multi-head attention based global pooling layer that captures the interaction between nodes according to their structural dependencies. We show that GMT satisfies both injectiveness and permutation invariance, such that it is at most as powerful as the Weisfeiler-Lehman graph isomorphism test. Moreover, our methods can be easily extended to the previous node clustering approaches for hierarchical graph pooling. Our experimental results show that GMT significantly outperforms state-of-the-art graph pooling methods on graph classification benchmarks with high memory and time efficiency, and obtains even larger performance gain on graph reconstruction and generation tasks.

Sequential Reptile_Inter-Task Gradient Alignment for Multilingual LearningMLAI2

Multilingual models jointly pretrained on multiple languages have achieved remarkable performance on various multilingual downstream tasks. Moreover, models finetuned on a single monolingual downstream task have shown to generalize to unseen languages. In this paper, we first show that it is crucial for those tasks to align gradients between them in order to maximize knowledge transfer while minimizing negative transfer. Despite its importance, the existing methods for gradient alignment either have a completely different purpose, ignore inter-task alignment, or aim to solve continual learning problems in rather inefficient ways. As a result of the misaligned gradients between tasks, the model suffers from severe negative transfer in the form of catastrophic forgetting of the knowledge acquired from the pretraining. To overcome the limitations, we propose a simple yet effective method that can efficiently align gradients between tasks. Specifically, we perform each inner-optimization by sequentially sampling batches from all the tasks, followed by a Reptile outer update. Thanks to the gradients aligned between tasks by our method, the model becomes less vulnerable to negative transfer and catastrophic forgetting. We extensively validate our method on various multi-task learning and zero-shot cross-lingual transfer tasks, where our method largely outperforms all the relevant baselines we consider.

KARNAUGH MAP using OpenGL (KMAP)Sagar Uday Kumar

The Gaussian Process Latent Variable Model (GPLVM)James McMurray

This document provides an outline for a talk on Gaussian Process Latent Variable Models (GPLVM). It begins with an introduction to why latent variable models are useful for dimensionality reduction. It then defines latent variable models and shows their graphical model representation. The document reviews PCA and introduces probabilistic versions like Probabilistic PCA (PPCA) and Dual PPCA. It describes how GPLVM generalizes these approaches using Gaussian processes. Examples applying GPLVM to face and motion data are provided, along with practical tips and an overview of GPLVM variants.

Masters ThesisMatt Moynihan

This document discusses Kinect fusion-based simultaneous localization and mapping (SLAM) systems for mobile robotics. It presents an example Kinect-based SLAM system that uses Kinect fusion to provide dense, real-time 3D mapping of a volume. The system aims to map larger volumes while also tracking human targets. It compares several variants of the iterative closest point (ICP) algorithm used for point cloud alignment, including ones using constant velocity and non-linear least squares estimation models. It also explores using RGB and depth data with features to initialize ICP transformations. Results show the frame-wise errors of different ICP variants on benchmark datasets.

Karnaugh map or K-map methodAbdullah Moin

Reduced Complexity Transfer Function Computation for Complex Indoor Channels ...Ramoni Adeogun, PhD

This slide set present our work on channel modelling for multi-room complex environments. The content has also been presented at and publish in IEEE PIMRC, 2018. The paper titled Transfer Function Computation for Complex Indoor Channels Using Propagation Graphs can be downloaded here: https://vbn.aau.dk/en/publications/transfer-function-computation-for-complex-indoor-channels-using-p

Double PatterningDanny Luk

Double patterning lithography is a technique used to print integrated circuit designs when feature sizes shrink below the resolution limits of a single exposure. It involves splitting the circuit layout into two masks and exposing the photo-resist layer twice to print the full design. Decomposing the circuit layout and assigning patterns to the two masks is an NP-hard graph coloring problem. The document describes techniques for decomposing the conflict graph that represents incompatible patterns, including using SPQR trees to decompose into tri-connected components and solving each independently. Experimental results show the proposed method can achieve a 3-10x speedup over other approaches.

Dimension Reduction And Visualization Of Large High Dimensional Data Via Inte...wl820609

This document discusses dimension reduction techniques for visualizing large, high-dimensional data. It presents multidimensional scaling (MDS) and generative topographic mapping (GTM) for this task. To address challenges of data size, an interpolation approach is introduced that maps new data points based on a reduced set of sample points. Experimental results show MDS and GTM interpolation can efficiently visualize millions of data points in 2-3 dimensions with reasonable quality compared to processing all points directly.

Double Patterning (4/2 update)Danny Luk

This document discusses double patterning lithography techniques. It introduces how optical lithography is approaching its limits and double patterning is needed for smaller feature sizes. It describes the double patterning process and challenges including feature distortion and decreased yield. The document outlines techniques for polygon cutting, priority search trees, and decomposing conflict graphs into tri-connected components to solve the layout splitting problem. Experimental results on test cases including a 320k polygon design show the method achieves 3-10x speedup.

Multiple region of interest tracking of non rigid objects using demon's algor...csandit

In this paper we propose an algorithm for tracking multiple ROI (region of interest) undergoing non-rigid transformations. Demon's algorithm based on the idea of Maxwell's demon, has been applied here to estimate the displacement field for tracking of multiple ROI. This algorithm works on pixel intensities of the sequence of images thus making it suitable for tracking objects/regions undergoing non-rigid transformations. We have incorporated a pyramid-based approach for demon's algorithm computations of displacement field, which leads to significant reduction in the convergence speed and improvement in the accuracy. This algorithm is applied for tracking non-rigid objects in laproscopy videos which would aid surgeons in Minimal Invasive Surgery (MIS).

MULTIPLE REGION OF INTEREST TRACKING OF NON-RIGID OBJECTS USING DEMON'S ALGOR...cscpconf

1) The document proposes a multiple region of interest (ROI) tracking algorithm for non-rigid objects using Demon's algorithm and a pyramidal approach. 2) A pyramidal implementation of Demon's algorithm improves computational efficiency and accuracy of tracking by calculating displacement fields at multiple image resolutions. 3) The algorithm is applied to track non-rigid ROIs in laparoscopy videos, which could help surgeons perform minimal invasive surgery.

HDRF: Stream-Based Partitioning for Power-Law GraphsFabio Petroni, PhD

F. Petroni, L. Querzoni, K. Daudjee, S. Kamali and G. Iacoboni: "HDRF: Stream-Based Partitioning for Power-Law Graphs." In: Proceedings of the 24th ACM International Conference on Information and Knowledge Management (CIKM), 2015. Abstract: "Balanced graph partitioning is a fundamental problem that is receiving growing attention with the emergence of distributed graph-computing (DGC) frameworks. In these frameworks, the partitioning strategy plays an important role since it drives the communication cost and the workload balance among computing nodes, thereby affecting system performance. However, existing solutions only partially exploit a key characteristic of natural graphs commonly found in the real-world: their highly skewed power-law degree distributions. In this paper, we propose High-Degree (are) Replicated First (HDRF), a novel streaming vertex-cut graph partitioning algorithm that effectively exploits skewed degree distributions by explicitly taking into account vertex degree in the placement decision. We analytically and experimentally evaluate HDRF on both synthetic and real-world graphs and show that it outperforms all existing algorithms in partitioning quality."

LCBM: Statistics-Based Parallel Collaborative FilteringFabio Petroni, PhD

F. Petroni, L. Querzoni, R. Beraldi, M. Paolucci: "LCBM: Statistics-Based Parallel Collaborative Filtering." In: Proceedings of the 17th International Conference on Business Information Systems (BIS), 2014. Abstract: "In the last ten years, recommendation systems evolved from novelties to powerful business tools, deeply changing the internet industry. Collaborative Filtering (CF) represents today’s a widely adopted strategy to build recommendation engines. The most advanced CF techniques (i.e. those based on matrix factorization) provide high quality results, but may incur prohibitive computational costs when applied to very large data sets. In this paper we present Linear Classifier of Beta distributions Means (LCBM), a novel collaborative filtering algorithm for binary ratings that is (i) inherently parallelizable and (ii) provides results whose quality is on-par with state-of-the-art solutions (iii) at a fraction of the computational cost."

Object Detection Beyond Mask R-CNN and RetinaNet IIIWanjin Yu

This document provides an overview of fine-grained image analysis. It begins with background on computer vision, deep learning, and traditional image recognition/retrieval. It then introduces fine-grained image analysis, distinguishing it from generic image recognition through examples. Challenges of fine-grained analysis are discussed, including small inter-class variance and large intra-class variance. Real-world applications of fine-grained analysis are presented across domains like species identification.

Kernel Descriptors for Visual RecognitionPriyatham Bollimpalli

The paper proposes novel kernel descriptors for visual recognition based on gradient, color, and local binary pattern (shape) features. Kernel descriptors reduce granularity of pixel features and better capture image variations compared to existing methods like SIFT. Gradient kernel descriptor performed best on four datasets for image classification, outperforming SIFT and other methods. The descriptors provide a computationally feasible way to learn high-level visual features using kernel methods.

Aerial detection part2ssuser456ad6

Analysis of Impact of Graph Theory in Computer ApplicationIRJET Journal

This document discusses several applications of graph theory in computer science. It summarizes how graph theory is used in map coloring, mobile phone networks, computer network security, modeling ad-hoc networks, fault tolerant computing systems, and clustering web documents. Graph theory provides structural models that can represent problems in these domains and enable new algorithms and solutions. Key applications mentioned include using graph coloring for frequency assignment in mobile networks, modeling network topology for worm propagation analysis, and representing documents and their relationships as graphs for clustering. Overall, the document outlines how graph theoretical concepts and methodologies are widely utilized to solve problems in computer science research areas.

Double PatterningDanny Luk

Double patterning lithography is a technique used to overcome the limitations of optical lithography. It works by splitting the mask pattern into two separate exposures to reduce feature density. The document describes the key techniques used in their software to solve the layout splitting problem, including a novel polygon cutting algorithm, dynamic priority search trees, and representing the problem as a tri-connected graph to decompose it into independent subproblems. Experimental results showed their method achieved a 3-10x speedup over other approaches.

computervision projectLianli Liu

The document summarizes and compares three recent face alignment algorithms: supervised descent method (SDM), local binary features regression (LBF), and face alignment under poses and occlusion (HPO). SDM formulates alignment as regression of local image features to landmark movements. LBF learns local binary features from regions around landmarks and uses global regression. HPO predicts landmark visibility and learns descent directions considering occlusion. The document evaluates the methods on standard datasets, finding SDM and LBF perform well except under extreme conditions, while HPO better handles poses and occlusion.

DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...Artem Lutov

Slides of the presentation given at BigData'19, special session on Information Granulation in Data Science and Scalable Computing. The fully automatic (i.e., without any manual tuning) graph embedding (i.e., network representation learning, unsupervised feature extraction) performed in near-linear time is presented. The resulting embeddings are interpretable, preserve both low- and high-order structural proximity of the graph nodes, computed (i.e., learned) by orders of magnitude faster and perform competitively to the manually tuned best state-of-the-art embedding techniques evaluated on diverse tasks of graph analysis.

Comparison of Various RCNN techniques for Classification of Object from ImageIRJET Journal

The document compares various region-based convolutional neural network (RCNN) techniques for classifying objects in images, including RCNN, Fast RCNN, and Faster RCNN. It first provides background on CNNs and how they are used for computer vision tasks like object recognition. It then describes the basic RCNN technique which uses selective search to generate region proposals that are input to a CNN for feature extraction and an SVM for classification. Fast RCNN improves on this by using a region of interest pooling layer to process features from the CNN in parallel rather than through separate layers. Faster RCNN introduces a region proposal network to generate proposals within the network to speed up processing. The techniques are implemented and evaluated on the CALTECH-101

Poster2013xinhuima

Dr. Xinhui Ma's research focuses on local refinement of T-splines, NURBS compatible subdivision surfaces, and adaptive finite element methods. He aims to bring together the flexibility of subdivision surfaces and the accuracy of NURBS for engineering and entertainment applications. His work also involves developing efficient local refinement approaches for T-splines to improve design and simulation speed.

GRAPH MATCHING ALGORITHM FOR TASK ASSIGNMENT PROBLEMIJCSEA Journal

Task assignment is one of the most challenging problems in distributed computing environment. An optimal task assignment guarantees minimum turnaround time for a given architecture. Several approaches of optimal task assignment have been proposed by various researchers ranging from graph partitioning based tools to heuristic graph matching. Using heuristic graph matching, it is often impossible to get optimal task assignment for practical test cases within an acceptable time limit. In this paper, we have parallelized the basic heuristic graph-matching algorithm of task assignment which is suitable only for cases where processors and inter processor links are homogeneous. This proposal is a derivative of the basic task assignment methodology using heuristic graph matching. The results show that near optimal assignments are obtained much faster than the sequential program in all the cases with reasonable speed-up.

MATLAB IMPLEMENTATION OF SELF-ORGANIZING MAPS FOR CLUSTERING OF REMOTE SENSIN...Daksh Raj Chopra

Deep Graph Contrastive Representation Learning.pptxssuser2624f71

This document summarizes a research paper on graph contrastive representation learning (GRACE) using an unsupervised framework. GRACE generates two graph views through random corruption, then trains a model with a contrastive loss to maximize agreement between node embeddings in the two views. It considers corruption at both the topology and node attribute levels. Experiments on citation networks show GRACE achieves competitive performance for transductive and inductive node classification tasks.

最近の研究情勢についていくために - Deep Learningを中心に - Hiroshi Fukui

This document summarizes key developments in deep learning for object detection from 2012 onwards. It begins with a timeline showing that 2012 was a turning point, as deep learning achieved record-breaking results in image classification. The document then provides overviews of 250+ contributions relating to object detection frameworks, fundamental problems addressed, evaluation benchmarks and metrics, and state-of-the-art performance. Promising future research directions are also identified.

IRJET - Object Detection using Deep Learning with OpenCV and PythonIRJET Journal

This document summarizes research on object detection techniques using deep learning. It discusses using the YOLO algorithm to identify objects in images using a single neural network that predicts bounding boxes and class probabilities. The document reviews prior research on algorithms like R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN and RetinaNet. It then describes the YOLO loss function and methodology for finding bounding boxes of objects in an image. The document concludes that YOLO is well-suited for real-time object detection applications due to its advantages over other algorithms.

Automatic Differentiation and SciML in Reality: What can go wrong, and what t...Chris Rackauckas

How does automatic differentiation work, what happens when you apply it to equation solvers, and how can it go wrong? This talk is all about the details of how scientific machine learning (SciML) works. It goes into detail as to how neural networks are trained in the context of equation solvers, along with the numerical issues that can arise in the differentiation processes. https://sciml.ai/

More Related Content

What's hot (18)

Double PatterningDanny Luk

Dimension Reduction And Visualization Of Large High Dimensional Data Via Inte...wl820609

Double Patterning (4/2 update)Danny Luk

Multiple region of interest tracking of non rigid objects using demon's algor...csandit

MULTIPLE REGION OF INTEREST TRACKING OF NON-RIGID OBJECTS USING DEMON'S ALGOR...cscpconf

HDRF: Stream-Based Partitioning for Power-Law GraphsFabio Petroni, PhD

LCBM: Statistics-Based Parallel Collaborative FilteringFabio Petroni, PhD

Object Detection Beyond Mask R-CNN and RetinaNet IIIWanjin Yu

Kernel Descriptors for Visual RecognitionPriyatham Bollimpalli

Aerial detection part2ssuser456ad6

Analysis of Impact of Graph Theory in Computer ApplicationIRJET Journal

Double PatterningDanny Luk

computervision projectLianli Liu

DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...Artem Lutov

Comparison of Various RCNN techniques for Classification of Object from ImageIRJET Journal

Poster2013xinhuima

GRAPH MATCHING ALGORITHM FOR TASK ASSIGNMENT PROBLEMIJCSEA Journal

MATLAB IMPLEMENTATION OF SELF-ORGANIZING MAPS FOR CLUSTERING OF REMOTE SENSIN...Daksh Raj Chopra

Double PatterningDanny Luk

Dimension Reduction And Visualization Of Large High Dimensional Data Via Inte...wl820609

Double Patterning (4/2 update)Danny Luk

Multiple region of interest tracking of non rigid objects using demon's algor...csandit

MULTIPLE REGION OF INTEREST TRACKING OF NON-RIGID OBJECTS USING DEMON'S ALGOR...cscpconf

HDRF: Stream-Based Partitioning for Power-Law GraphsFabio Petroni, PhD

LCBM: Statistics-Based Parallel Collaborative FilteringFabio Petroni, PhD

Object Detection Beyond Mask R-CNN and RetinaNet IIIWanjin Yu

Kernel Descriptors for Visual RecognitionPriyatham Bollimpalli

Aerial detection part2ssuser456ad6

Analysis of Impact of Graph Theory in Computer ApplicationIRJET Journal

Double PatterningDanny Luk

computervision projectLianli Liu

DAOR - Bridging the Gap between Community and Node Representations: Graph Emb...Artem Lutov

Comparison of Various RCNN techniques for Classification of Object from ImageIRJET Journal

Poster2013xinhuima

GRAPH MATCHING ALGORITHM FOR TASK ASSIGNMENT PROBLEMIJCSEA Journal

MATLAB IMPLEMENTATION OF SELF-ORGANIZING MAPS FOR CLUSTERING OF REMOTE SENSIN...Daksh Raj Chopra

Similar to Learning to Extrapolate Knowledge: Transductive Few-shot Out-of-Graph Link Prediction (20)

Deep Graph Contrastive Representation Learning.pptxssuser2624f71

最近の研究情勢についていくために - Deep Learningを中心に - Hiroshi Fukui

IRJET - Object Detection using Deep Learning with OpenCV and PythonIRJET Journal

Automatic Differentiation and SciML in Reality: What can go wrong, and what t...Chris Rackauckas

self operating mapsAltafSMT

This document is the presentation slides for a technical seminar on Self Organizing Maps. It introduces self-organizing maps as an unsupervised neural network technique. The presentation covers topics such as statistical pattern recognition, unsupervised pattern classification, advantages and disadvantages of self-organizing maps, and concludes that self-organizing maps provide a structural representation of input data.

A simple framework for contrastive learning of visual representationsDevansh16

Link: https://machine-learning-made-simple.medium.com/learnings-from-simclr-a-framework-contrastive-learning-for-visual-representations-6c145a5d8e99 If you'd like to discuss something, text me on LinkedIn, IG, or Twitter. To support me, please use my referral link to Robinhood. It's completely free, and we both get a free stock. Not using it is literally losing out on free money. Check out my other articles on Medium. : https://rb.gy/zn1aiu My YouTube: https://rb.gy/88iwdd Reach out to me on LinkedIn. Let's connect: https://rb.gy/m5ok2y My Instagram: https://rb.gy/gmvuy9 My Twitter: https://twitter.com/Machine01776819 My Substack: https://devanshacc.substack.com/ Live conversations at twitch here: https://rb.gy/zlhk9y Get a free stock on Robinhood: https://join.robinhood.com/fnud75 This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels. Comments: ICML'2020. Code and pretrained models at this https URL Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML) Cite as: arXiv:2002.05709 [cs.LG] (or arXiv:2002.05709v3 [cs.LG] for this version) Submission history From: Ting Chen [view email] [v1] Thu, 13 Feb 2020 18:50:45 UTC (5,093 KB) [v2] Mon, 30 Mar 2020 15:32:51 UTC (5,047 KB) [v3] Wed, 1 Jul 2020 00:09:08 UTC (5,829 KB)

Introduction Of Artificial neural networkNagarajan

The document summarizes different types of artificial neural networks including their structure, learning paradigms, and learning rules. It discusses artificial neural networks (ANN), their advantages, and major learning paradigms - supervised, unsupervised, and reinforcement learning. It also explains different mathematical synaptic modification rules like backpropagation of error, correlative Hebbian, and temporally-asymmetric Hebbian learning rules. Specific learning rules discussed include the delta rule, the pattern associator, and the Hebb rule.

NS - CUK Seminar: V.T.Hoang, Review on "Long Range Graph Benchmark.", NeurIPS...ssuser4b1f48

The document proposes new graph datasets and benchmarks for evaluating long-range modeling in graph neural networks. It introduces 5 new datasets that have large graph sizes and tasks requiring distant node interactions. Experiments show that local graph neural networks perform poorly on these datasets while graph transformers with long-range capabilities perform better, indicating the datasets effectively measure long-range modeling ability. The datasets are intended to advance research on addressing the long-range information bottleneck in graph neural networks.

Model Evaluation in the land of Deep LearningPramit Choudhary

1. The document discusses model interpretation and techniques for interpreting machine learning models, especially deep neural networks. 2. It describes what model interpretation is, its importance and benefits, and provides examples of interpretability algorithms like dimensionality reduction, manifold learning, and visualization techniques. 3. The document aims to help make machine learning models more transparent and understandable to humans in order to build trust and improve model evaluation, debugging and feature engineering.

[20240825_LabSeminar_Huy]Self-Supervised Learning for Multilevel Skeleton-Bas...thanhdowork

Deep learning and computer visionMeetupDataScienceRoma

Uncovering the Structural Fairness in Graph Contrastive Learning.pptxssuser2624f71

This document presents a method called GRADE for improving structural fairness in graph contrastive learning (GCL) models. GRADE aims to increase connections within communities and decrease connections between communities for tail nodes, while purifying the neighborhoods of head nodes. It does this through graph augmentation techniques during training. Experiments on citation and social networks demonstrate that GRADE outperforms baseline GCL models in node classification tasks while enhancing fairness to degree bias. However, GRADE's heuristic design leaves room for future theoretical improvements.

Anomaly Detection for Real-World SystemsManojit Nandi

(1) Anomaly detection aims to identify data points that are noticeably different from expected patterns in a dataset. (2) Common approaches include statistical modeling, machine learning classification, and algorithms designed specifically for anomaly detection. (3) Streaming data poses unique challenges due to limited memory and need for rapid identification of anomalies. (4) Heuristics like z-scores and median absolute deviation provide robust ways to measure how extreme observations are compared to a distribution's center. (5) Density-based methods quantify how isolated data points are to identify anomalies. (6) Time series algorithms decompose trends and seasonality to identify global and local anomalous spikes and troughs.

NIPS2007: deep belief netszukun

- Geoffrey Hinton gives a tutorial on deep belief nets and how to learn multi-layer generative models of unlabeled data by learning one layer of features at a time using restricted Boltzmann machines (RBMs). - RBMs make it possible to efficiently learn deep generative models one layer at a time by approximating the intractable posterior distribution over hidden units given visible data. - Layer-by-layer unsupervised pre-training of features followed by discriminative fine-tuning improves classification performance on benchmark datasets like MNIST compared to backpropagation alone.

Colloquium.pptxMythili680896

This document provides an introduction to graph neural networks (GNNs). It discusses that GNNs are designed to perform inference on graph data using neural networks. GNNs can analyze graph data by performing node classification, graph classification, graph visualization, link prediction, and graph clustering. The document explains that GNNs apply the concepts of node embedding and message passing to learn node representations that encode structural information from a graph. It describes how GNNs use neural network layers and aggregation functions to learn representations by propagating and transforming node feature information across a graph.

graph_embeddingsArmando Vieira

This document discusses using convolutional neural networks and self-organized maps to visualize knowledge graphs. It presents a compositional model for embedding nodes and relationships in a knowledge graph into a vector space. A self-organized map is used to cluster the embeddings and extract semantic fingerprints. The fingerprints are useful for knowledge discovery and classification tasks. The technique is applied to a subset of the CTD knowledge graph containing compound-gene/protein interactions, and the results are comparable to structural models.

Machine learning and vulnerabilitiesgalazzo

This document discusses machine learning vulnerabilities and adversarial attacks. It begins by introducing perceptrons and different machine learning approaches like logistic regression, support vector machines, and neural networks. It then describes how convolutional neural networks can be used for image classification but are vulnerable to adversarial examples crafted by subtly tweaking pixel values. The document outlines the steps to generate adversarial examples and provides a Python code snippet. It notes that including adversarial examples in training can help defenses.

NS-CUK Seminar: S.T.Nguyen, Review on "Improving Graph Neural Network Express...ssuser4b1f48

（DL輪読）Matching Networks for One Shot LearningMasahiro Suzuki

1. Matching Networks is a neural network architecture proposed by DeepMind for one-shot learning. 2. The network learns to classify novel examples by comparing them to a small support set of examples, using an attention mechanism to focus on the most relevant support examples. 3. The network is trained using a meta-learning approach, where it learns to learn from small support sets to classify novel examples from classes not seen during training.

250331_Thuy_Labseminar[Higher Order Structures For Graph Explanations].pptxthanhdowork

Deep Graph Contrastive Representation Learning.pptxssuser2624f71

最近の研究情勢についていくために - Deep Learningを中心に - Hiroshi Fukui

IRJET - Object Detection using Deep Learning with OpenCV and PythonIRJET Journal

Automatic Differentiation and SciML in Reality: What can go wrong, and what t...Chris Rackauckas

self operating mapsAltafSMT

A simple framework for contrastive learning of visual representationsDevansh16

Introduction Of Artificial neural networkNagarajan

NS - CUK Seminar: V.T.Hoang, Review on "Long Range Graph Benchmark.", NeurIPS...ssuser4b1f48

Model Evaluation in the land of Deep LearningPramit Choudhary

[20240825_LabSeminar_Huy]Self-Supervised Learning for Multilevel Skeleton-Bas...thanhdowork

Deep learning and computer visionMeetupDataScienceRoma

Uncovering the Structural Fairness in Graph Contrastive Learning.pptxssuser2624f71

Anomaly Detection for Real-World SystemsManojit Nandi

NIPS2007: deep belief netszukun

Colloquium.pptxMythili680896

graph_embeddingsArmando Vieira

Machine learning and vulnerabilitiesgalazzo

NS-CUK Seminar: S.T.Nguyen, Review on "Improving Graph Neural Network Express...ssuser4b1f48

（DL輪読）Matching Networks for One Shot LearningMasahiro Suzuki

250331_Thuy_Labseminar[Higher Order Structures For Graph Explanations].pptxthanhdowork

More from MLAI2 (20)

Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Unce...MLAI2

Numerous recent works utilize bi-Lipschitz regularization of neural network layers to preserve relative distances between data instances in the feature spaces of each layer. This distance sensitivity with respect to the data aids in tasks such as uncertainty calibration and out-of-distribution (OOD) detection. In previous works, features extracted with a distance sensitive model are used to construct feature covariance matrices which are used in deterministic uncertainty estimation or OOD detection. However, in cases where there is a distribution over tasks, these methods result in covariances which are sub-optimal, as they may not leverage all of the meta information which can be shared among tasks. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. Additionally, we propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution which is well calibrated under a distributional dataset shift.

Online Hyperparameter Meta-Learning with Hypergradient DistillationMLAI2

Many gradient-based meta-learning methods assume a set of parameters that do not participate in inner-optimization, which can be considered as hyperparameters. Although such hyperparameters can be optimized using the existing gradient-based hyperparameter optimization (HO) methods, they suffer from the following issues. Unrolled differentiation methods do not scale well to high-dimensional hyperparameters or horizon length, Implicit Function Theorem (IFT) based methods are restrictive for online optimization, and short horizon approximations suffer from short horizon bias. In this work, we propose a novel HO method that can overcome these limitations, by approximating the second-order term with knowledge distillation. Specifically, we parameterize a single Jacobian-vector product (JVP) for each HO step and minimize the distance from the true second-order term. Our method allows online optimization and also is scalable to the hyperparameter dimension and the horizon length. We demonstrate the effectiveness of our method on two different meta-learning methods and three benchmark datasets.

Online Coreset Selection for Rehearsal-based Continual LearningMLAI2

A dataset is a shred of crucial evidence to describe a task. However, each data point in the dataset does not have the same potential, as some of the data points can be more representative or informative than others. This unequal importance among the data points may have a large impact in rehearsal-based continual learning, where we store a subset of the training examples (coreset) to be replayed later to alleviate catastrophic forgetting. In continual learning, the quality of the samples stored in the coreset directly affects the model's effectiveness and efficiency. The coreset selection problem becomes even more important under realistic settings, such as imbalanced continual learning or noisy data scenarios. To tackle this problem, we propose Online Coreset Selection (OCS), a simple yet effective method that selects the most representative and informative coreset at each iteration and trains them in an online manner. Our proposed method maximizes the model's adaptation to a target dataset while selecting high-affinity samples to past tasks, which directly inhibits catastrophic forgetting. We validate the effectiveness of our coreset selection mechanism over various standard, imbalanced, and noisy datasets against strong continual learning baselines, demonstrating that it improves task adaptation and prevents catastrophic forgetting in a sample-efficient manner.

Representational Continuity for Unsupervised Continual LearningMLAI2

Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (Lump), a simple yet effective technique that interpolates between the current task and previous tasks' instances to alleviate catastrophic forgetting for unsupervised representations.

Skill-Based Meta-Reinforcement LearningMLAI2

While deep reinforcement learning methods have shown impressive results in robot learning, their sample inefficiency makes the learning of complex, long-horizon behaviors with real robot systems infeasible. To mitigate this issue, meta-reinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn. Yet, the application has been limited to short-horizon tasks with dense rewards. To enable learning long-horizon behaviors, recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations. While these approaches yield improved sample efficiency, millions of interactions with environments are still required to solve complex tasks. In this work, we devise a method that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning. Specifically, we propose to (1) extract reusable skills and a skill prior from offline datasets, (2) meta-train a high-level policy that learns to efficiently compose learned skills into long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve an unseen target task. Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve long-horizon novel target tasks by combining the strengths of meta-learning and the usage of offline datasets, while prior approaches in RL, meta-RL, and multi-task RL require substantially more environment interactions to solve the tasks.

Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Genera...MLAI2

1) The document presents a method called FREED that uses fragment-based molecule generation guided by reinforcement learning to discover novel drug hits. 2) FREED explicitly constrains molecule generation to pharmacologically acceptable fragments to avoid toxic structures, which is more effective than implicit constraint methods. 3) FREED's exploratory RL algorithm prioritizes experience replay to encourage visiting novel states and finding diverse optima in the constrained chemical space.

Mini-Batch Consistent Slot Set Encoder For Scalable Set EncodingMLAI2

Most existing set encoding algorithms operate under the implicit assumption that all the set elements are accessible, and that there are ample computational and memory resources to load the set into memory during training and inference. However, both assumptions fail when the set is excessively large such that it is impossible to load all set elements into memory, or when data arrives in a stream. To tackle such practical challenges in large-scale set encoding, the general set-function constraints of permutation invariance and equivariance are not sufficient. We introduce a new property termed Mini-Batch Consistency (MBC) that is required for large scale mini-batch set encoding. Additionally, we present a scalable and efficient attention-based set encoding mechanism that is amenable to mini-batch processing of sets, and capable of updating set representations as data arrives. The proposed method adheres to the required symmetries of invariance and equivariance as well as maintaining MBC for any partition of the input set. We perform extensive experiments and show that our method is computationally efficient and results in rich set encoding representations for set-structured data.

Task Adaptive Neural Network Search with Meta-Contrastive LearningMLAI2

Most conventional Neural Architecture Search (NAS) approaches are limited in that they only generate architectures without searching for the optimal parameters. While some NAS methods handle this issue by utilizing a supernet trained on a large-scale dataset such as ImageNet, they may be suboptimal if the target tasks are highly dissimilar from the dataset the supernet is trained on. To address such limitations, we introduce a novel problem of Neural Network Search (NNS), whose goal is to search for the optimal pretrained network for a novel dataset and constraints (e.g. number of parameters), from a model zoo. Then, we propose a novel framework to tackle the problem, namely Task-Adaptive Neural Network Search (TANS). Given a model-zoo that consists of network pretrained on diverse datasets, we use a novel amortized meta-learning framework to learn a cross-modal latent space with contrastive loss, to maximize the similarity between a dataset and a high-performing network on it, and minimize the similarity between irrelevant dataset-network pairs. We validate the effectiveness and efficiency of our method on ten real-world datasets, against existing NAS/AutoML baselines. The results show that our method instantly retrieves networks that outperform models obtained with the baselines with significantly fewer training steps to reach the target performance, thus minimizing the total cost of obtaining a task-optimal network. Our code and the model-zoo are available at https://anonymous.4open.science/r/TANS-33D6

Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint L...MLAI2

While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning (FSSL). In this work, we study two essential scenarios of FSSL based on the location of the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server). We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning.

Meta-GMVAE: Mixture of Gaussian VAE for Unsupervised Meta-LearningMLAI2

Unsupervised learning aims to learn meaningful representations from unlabeled data which can captures its intrinsic structure, that can be transferred to downstream tasks. Meta-learning, whose objective is to learn to generalize across tasks such that the learned model can rapidly adapt to a novel task, shares the spirit of unsupervised learning in that the both seek to learn more effective and efficient learning procedure than learning from scratch. The fundamental difference of the two is that the most meta-learning approaches are supervised, assuming full access to the labels. However, acquiring labeled dataset for meta-training not only is costly as it requires human efforts in labeling but also limits its applications to pre-defined task distributions. In this paper, we propose a principled unsupervised meta-learning model, namely Meta-GMVAE, based on Variational Autoencoder (VAE) and set-level variational inference. Moreover, we introduce a mixture of Gaussian (GMM) prior, assuming that each modality represents each class-concept in a randomly sampled episode, which we optimize with Expectation-Maximization (EM). Then, the learned model can be used for downstream few-shot classification tasks, where we obtain task-specific parameters by performing semi-supervised EM on the latent representations of the support and query set, and predict labels of the query set by computing aggregated posteriors. We validate our model on Omniglot and Mini-ImageNet datasets by evaluating its performance on downstream few-shot classification tasks. The results show that our model obtain impressive performance gains over existing unsupervised meta-learning baselines, even outperforming supervised MAML on a certain setting.

Contrastive Learning with Adversarial Perturbations for Conditional Text Gene...MLAI2

Recently, sequence-to-sequence (seq2seq) models with the Transformer architecture have achieved remarkable performance on various conditional text generation tasks, such as machine translation. However, most of them are trained with teacher forcing with the ground truth label given at each time step, without being exposed to incorrectly generated tokens during training, which hurts its generalization to unseen inputs, that is known as the “exposure bias” problem. In this work, we propose to mitigate the conditional text generation problem by contrasting positive pairs with negative pairs, such that the model is exposed to various valid or incorrect perturbations of the inputs, for improved generalization. However, training the model with naïve contrastive learning framework using random non-target sequences as negative examples is suboptimal, since they are easily distinguishable from the correct output, especially so with models pretrained with large text corpora. Also, generating positive examples requires domain-specific augmentation heuristics which may not generalize over diverse domains. To tackle this problem, we propose a principled method to generate positive and negative samples for contrastive learning of seq2seq models. Specifically, we generate negative examples by adding small perturbations to the input sequence to minimize its conditional likelihood, and positive examples by adding large perturbations while enforcing it to have a high conditional likelihood. Such “hard” positive and negative pairs generated using our method guides the model to better distinguish correct outputs from incorrect ones. We empirically show that our proposed method significantly improves the generalization of the seq2seq on three text generation tasks — machine translation, text summarization, and question generation.

Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Le...MLAI2

Although recent multi-task learning methods have shown to be effective in improving the generalization of deep neural networks, they should be used with caution for safety-critical applications, such as clinical risk prediction. This is because even if they achieve improved task-average performance, they may still yield degraded performance on individual tasks, which may be critical (e.g., prediction of mortality risk). Existing asymmetric multi-task learning methods tackle this negative transfer problem by performing knowledge transfer from tasks with low loss to tasks with high loss. However, using loss as a measure of reliability is risky since it could be a result of overfitting. In the case of time-series prediction tasks, knowledge learned for one task (e.g., predicting the sepsis onset) at a specific timestep may be useful for learning another task (e.g., prediction of mortality) at a later timestep, but lack of loss at each timestep makes it difficult to measure the reliability at each timestep. To capture such dynamically changing asymmetric relationships between tasks in time-series data, we propose a novel temporal asymmetric multi-task learning model that performs knowledge transfer from certain tasks/timesteps to relevant uncertain tasks, based on feature-level uncertainty. We validate our model on multiple clinical risk prediction tasks against various deep learning models for time-series prediction, which our model significantly outperforms, without any sign of negative transfer. Further qualitative analysis of learned knowledge graphs by clinicians shows that they are helpful in analyzing the predictions of the model. Our final code is available at this https://github.com/anhtuan5696/TPAMTL.

MetaPerturb: Transferable Regularizer for Heterogeneous Tasks and ArchitecturesMLAI2

MetaPerturb is a meta-learned perturbation function that can enhance generalization of neural networks on different tasks and architectures. It proposes a novel meta-learning framework involving jointly training a main model and perturbation module on multiple source tasks to learn a transferable perturbation function. This meta-learned perturbation function can then be transferred to improve performance of a target model on an unseen target task or architecture, outperforming baselines on various datasets and architectures.

Adversarial Self-Supervised Contrastive LearningMLAI2

Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions, which are then used to augment the training of the model for improved robustness. While some recent works propose semi-supervised adversarial learning methods that utilize unlabeled data, they still require class labels. However, do we really need class labels at all, for adversarially robust training of deep neural networks? In this paper, we propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. Further, we present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data, which aims to maximize the similarity between a random augmentation of a data sample and its instance-wise adversarial perturbation. We validate our method, Robust Contrastive Learning (RoCL), on multiple benchmark datasets, on which it obtains comparable robust accuracy over state-of-the-art supervised adversarial learning methods, and significantly improved robustness against the black box and unseen types of attacks. Moreover, with further joint fine-tuning with supervised adversarial loss, RoCL obtains even higher robust accuracy over using self-supervised learning alone. Notably, RoCL also demonstrate impressive results in robust transfer learning.

Neural Mask Generator : Learning to Generate Adaptive WordMaskings for Langu...MLAI2

We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings.

Cost-effective Interactive Attention Learning with Neural Attention ProcessMLAI2

We propose a novel interactive learning framework which we refer to as Interactive Attention Learning (IAL), in which the human supervisors interactively manipulate the allocated attentions, to correct the model's behavior by updating the attention-generating network. However, such a model is prone to overfitting due to scarcity of human annotations, and requires costly retraining. Moreover, it is almost infeasible for the human annotators to examine attentions on tons of instances and features. We tackle these challenges by proposing a sample-efficient attention mechanism and a cost-effective reranking algorithm for instances and features. First, we propose Neural Attention Process (NAP), which is an attention generator that can update its behavior by incorporating new attention-level supervisions without any retraining. Secondly, we propose an algorithm which prioritizes the instances and the features by their negative impacts, such that the model can yield large improvements with minimal human feedback. We validate IAL on various time-series datasets from multiple domains (healthcare, real-estate, and computer vision) on which it significantly outperforms baselines with conventional attention mechanisms, or without cost-effective reranking, with substantially less retraining and human-model interaction cost.

Adversarial Neural Pruning with Latent Vulnerability SuppressionMLAI2

Despite the remarkable performance of deep neural networks on various computer vision tasks, they are known to be susceptible to adversarial perturbations, which makes it challenging to deploy them in real-world safety-critical applications. In this paper, we conjecture that the leading cause of adversarial vulnerability is the distortion in the latent feature space, and provide methods to suppress them effectively. Explicitly, we define vulnerability for each latent feature and then propose a new loss for adversarial learning, Vulnerability Suppression (VS) loss, that aims to minimize the feature-level vulnerability during training. We further propose a Bayesian framework to prune features with high vulnerability to reduce both vulnerability and loss on adversarial samples. We validate our Adversarial Neural Pruning with Vulnerability Suppression (ANP-VS) method on multiple benchmark datasets, on which it not only obtains state-of-the-art adversarial robustness but also improves the performance on clean examples, using only a fraction of the parameters used by the full network. Further qualitative analysis suggests that the improvements come from the suppression of feature-level vulnerability.

Generating Diverse and Consistent QA pairs from Contexts with Information-Max...MLAI2

One of the most crucial challenges in question answering (QA) is the scarcity of labeled data, since it is costly to obtain question-answer (QA) pairs for a target text domain with human annotation. An alternative approach to tackle the problem is to use automatically generated QA pairs from either the problem context or from large amount of unstructured texts (e.g. Wikipedia). In this work, we propose a hierarchical conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts, while maximizing the mutual information between generated QA pairs to ensure their consistency. We validate our Information Maximizing Hierarchical Conditional Variational AutoEncoder (Info-HCVAE) on several benchmark datasets by evaluating the performance of the QA model (BERT-base) using only the generated QA pairs (QA-based evaluation) or by using both the generated and human-labeled pairs (semi-supervised learning) for training, against state-of-the-art baseline models. The results show that our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.

Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distrib...MLAI2

While tasks could come with varying the number of instances and classes in realistic settings, the existing meta-learning approaches for few-shot classification assume that number of instances per task and class is fixed. Due to such restriction, they learn to equally utilize the meta-knowledge across all the tasks, even when the number of instances per task and class largely varies. Moreover, they do not consider distributional difference in unseen tasks, on which the meta-knowledge may have less usefulness depending on the task relatedness. To overcome these limitations, we propose a novel meta-learning model that adaptively balances the effect of the meta-learning and task-specific learning within each task. Through the learning of the balancing variables, we can decide whether to obtain a solution by relying on the meta-knowledge or task-specific learning. We formulate this objective into a Bayesian inference framework and tackle it using variational inference. We validate our Bayesian Task-Adaptive Meta-Learning (Bayesian TAML) on two realistic task- and class-imbalanced datasets, on which it significantly outperforms existing meta-learning approaches. Further ablation study confirms the effectiveness of each balancing component and the Bayesian learning framework.

Meta Dropout: Learning to Perturb Latent Features for Generalization MLAI2

A machine learning model that generalizes well should obtain low errors on unseen test examples. Thus, if we know how to optimally perturb training examples to account for test examples, we may achieve better generalization performance. However, obtaining such perturbation is not possible in standard machine learning frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose a novel regularization method, meta-dropout, which learns to perturb the latent features of training examples for generalization in a meta-learning framework. Specifically, we meta-learn a noise generator which outputs a multiplicative noise distribution for latent features, to obtain low errors on the test instances in an input-dependent manner. Then, the learned noise generator can perturb the training examples of unseen tasks at the meta-test time for improved generalization. We validate our method on few-shot classification datasets, whose results show that it significantly improves the generalization performance of the base model, and largely outperforms existing regularization methods such as information bottleneck, manifold mixup, and information dropout.