Design Space for Graph

Neural Network
孙佳傲 22121401

1. Introduction
The development of contemporary Graph Neural Networks (GNNs) is closely linked
to the revolution in deep learning. This revolution has significantly changed how we
learn from vast amounts of data, eliminating the need for manual feature engineering.
Now, we are on the cusp of the next wave in artificial intelligence development,
which involves applying deep learning to relational data, or graphs. Our aim is to
input network data into a GNN and, through several layers of non-linear
transformations, effectively predict node labels, forecast new links, or even make
predictions about the entire graph. Furthermore, we can also generate new graphs.

In recent years, there has been a surge in interest in the study of Graph Neural
Networks (GNNs). As shown in Figure 1, between 2018 and 2020, the term "Graph
Neural Networks" has seen a rapid increase in its proportion among the keywords of
many submitted papers, indicating a thriving field of research. Today, the proportion
of papers related to Graph Neural Networks is significantly higher compared to areas
such as meta-learning, self-supervised learning, and neural architecture search.
 Figure1.ICLR-Keyword Growth 2018-2020

2. Problem Statement
Given the success of Graph Neural Networks (GNNs) in many areas, a variety of
network architectures, tuning strategies, and improvement techniques have been
proposed. To a certain extent, as professionals and scientists in the field, we would
want to have the most advanced GNNs to tackle our current tasks.

However, there are many possible GNN architectures, and the best-performing GNN
design for one task may not be suitable for another. This means we have to try all
possible GNN design options and hope to find one that works well for a specific task.
So, can we build a system that can automatically predict good GNN designs? What
kind of neural network architecture is effective for a given dataset and prediction
task?

More specifically, the following research questions need to be addressed:

1． How to explore different designs of GNNs?

2． How to determine that a specially designed network with certain characteristics is
suitable for a specific set of tasks?
3． How to construct a system that can automatically predict good GNN designs?
4． For a given dataset and prediction task, what kind of neural network architecture
will be effective?
3. Main Point 1-Design Space
Jure Leskovec defines the design space of Graph Neural Networks (GNNs) at three
levels:

1. Intra-layer design: The design of an individual GNN layer.

2. Inter-layer design: How to connect GNN layers.

3. Learning setup: The parameters for machine learning (such as batch size, learning
rate, optimizer, and number of training epochs).

In this way, Jure Leskovec and others aim to define the broadest possible design space
that can encompass a wide range of design possibilities while still being manageable.
Ultimately, they considered a design space that includes 315,000 potential
computational architectures.

Figure2.single-layer GNN

As shown in Figure 2, we will first consider the design of a single-layer GNN; then,
we will think about how to combine these individual GNN layers into a larger GNN
network architecture; additionally, we should consider what kind of learning
configurations to use. Below, we will analyze these three factors one by one.
 Figure3.intra-layer design of a GNN

As shown in Figure 3, suppose we are considering four dimensions when looking at

the intra-layer design of a GNN. At this point, we need to decide on the message-
passing mechanism, the message-aggregation function, whether to use Dropout in the
linear layers of the GNN, and whether to apply batch normalization.

4.Main Point 2-Task Space

Currently, tasks are often classified into three categories: node-level, edge-level, and
graph-level predictions. While this method of classification seems logical, it is not
precise enough. To improve this, Jure Leskovec and colleagues have designed a "task
similarity matrix" that allows us to quickly identify similar tasks and models that
perform well on these tasks, enabling the transfer of model architectures across tasks.

For each task, which consists of a dataset and its prediction task, we select a set of
"anchor models." We then characterize the task by ranking the performance of these
anchor models. Essentially, we consider tasks with similar rankings (similar
performance rankings of anchor models) to be similar. In this case, Jure Leskovec and
others used 12 anchor models to measure the similarity between different tasks.

According to Jure Leskovec, we can construct a GNN task space. Given a specific
task and dataset, we train a set of anchor models to perform the task. We then quantify
the performance differences of the models on specific tasks through ranking.

Jure Leskovec and his team calculated the similarity between tasks using the Kendall
rank correlation coefficient based on the performance of the anchor models. As shown
in Figure 4, each point represents a different GNN task. In this example, they aimed to
quantify the similarity between all different models using just 12 anchor models. The
scatter plot on the right side of Figure 43 shows the true value of similarity on the y-
axis and the task similarity calculated using only 12 different anchor models on the x-
axis. The strong positive correlation between the two is evident. Therefore, we can
conclude that using just 12 different anchor models can approximate the true task
similarity quite well.

Figure4.Task similarity

Based on the above method, Jure Leskovec obtained the task space similarity matrix
as shown in Figure 5. They considered 32 different tasks, including various node, link,
and graph-level classification tasks. Each dot in the grid represents the similarity
between tasks (i.e., the Kendall rank correlation coefficient), ranging from -1 to 1. It
can be seen that the tasks in the top left corner of the matrix have a very high
similarity, meaning that their performance rankings on the anchor models are very
similar. Similarly, the tasks in the bottom right corner of the matrix also have a high
similarity.
 Figure5.Proposed task similarity

5.Main Point 3-Transfer between Different

Tasks
As illustrated in Figure 6, the task space of Graph Neural Networks (GNNs) can guide
the transfer of models between different tasks. This means that we can apply the best-
performing model from Task A to a highly similar Task B. In other words, if two tasks
have a high Kendall rank correlation coefficient, we can transfer the best model
architectures from these tasks to quickly identify a new optimal architecture for a
given task.
 Figure6.Transfer between Different Tasks

For instance, suppose we want to apply a certain method to the 'ogbg-molhiv' dataset
for graph classification on the OGB platform. This dataset is unique, being 20 times
larger than the average dataset and having a highly imbalanced data distribution (with
only 1.4% positive samples in the training dataset). We need to perform out-of-
distribution generalization for this dataset.

To achieve this, we ran 12 anchor models on this new dataset and ranked their
performance. Then, we calculated the similarity of the rankings obtained from these
12 anchor models for this task and the rankings of models used across 32 tasks in this
example. We identified the most similar models and transferred the best-performing
model to the new dataset.

This task space allows us to quickly identify excellent model architectures suitable for
new tasks. For example, if Task B is very different from the new dataset we are
interested in, while Task A is very similar to our dataset of interest, transferring the
best model from Task A to the new dataset results in excellent performance (with an
accuracy close to 0.79), even achieving the best current model performance (the
previous best accuracy was 0.771). However, if we use the best model from Task B,
which is very different from the new task, the model's performance is much poorer
(with an accuracy of 0.736).

By simply calculating the similarity between the new dataset and the existing tasks in
the task space, we can quickly identify the most similar tasks and transfer their best
models to the new dataset for training. In this way, we may achieve the best model
performance on datasets that we have never used before.

6.References
1. Kipf, T. N., & Welling, M. (2016). Semi-Supervised Classification with Graph
Convolutional Networks. arXiv preprint arXiv:1609.02907.

2. Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y.
(2017). Graph Attention Networks. arXiv preprint arXiv:1710.10903.

3. Hamilton, W. L., Ying, Z., & Leskovec, J. (2017). Inductive Representation

Learning on Large Graphs. In NIPS.

4. Gilmer, J., Schoenholz, S. S., Riley, P. F., Li, Z., & Song, J. J. (2017). Neural
Message Passing for Quantum Chemistry. In ICML.

5. Xu, K., Hu, W., Leskovec, J., & Jegelka, S. (2018). How Powerful are Graph
Neural Networks? In ICLR.

6. Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Yu, P. S. (2020). A
Comprehensive Survey on Graph Neural Networks. IEEE Transactions on
Neural Networks and Learning Systems.

7. Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., & Sun, M. (2020).
Graph Neural Networks: A Review of Methods and Applications. AI Open, 1,
57–81.

8. Wang, S., Li, B., Liu, X., & Wu, Y. (2020). Graph Isomorphism Network:
Message Passing for Graphs. In NeurIPS.

9. Chen, T., Ma, T., & Xiao, E. (2020). Graph Convolutional Networks for Text
Classification. In AAAI.

10. Li, G., Li, K., He, S., Chen, J., & Bu, J. (2020). Deeper Insights into Graph
Convolutional Networks for Semi-Supervised Learning. In AAAI.

11. Wu, S., Pan, S., Long, G., Zhang, C., & Yu, P. S. (2020). A Comprehensive
Survey on Graph Neural Networks: Theoretical Foundations, Applications,
and Open Challenges. arXiv preprint arXiv:2001.05549.

12. Zitnik, M., Agrawal, M., & Leskovec, J. (2018). Modeling Polypharmacy Side
Effects with Graph Convolutional Networks. Bioinformatics, 34(13), i457–
i466.