CS224W: Machine Learning with Graphs

Jure Leskovec, Stanford University

http://cs224w.stanford.edu
  The class meets Tue and Thu 1:30-3:00pm
Pacific Time in person
▪ Videos of the lectures will be recorded and posted
on Canvas
 Structure of lectures:
▪ 60-70 minutes of a lecture
▪ During this time you can ask questions
▪ 10-20 minutes of a live Q&A/discussion session at
the end of the lecture

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 2

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 3
 Date Topic Date Topic
1. Introduction; Machine Learning 11. Reasoning over Knowledge
Tue, Sep 21 Tue, Oct 26
for Graphs Graphs
2. Traditional Methods for ML on 12. Frequent Subgraph Mining
Thu, Sep 23 Thu, Oct 28
Graphs with GNNs
13. Community Structure in
Tue, Sep 28 3. Node Embeddings Thu, Nov 4
Networks
14. Traditional Generative
Thu, Sep 30 4. Link Analysis: PageRank Tue, Nov 9
Models for Graphs
5. Label Propagation for Node 15. Deep Generative Models for
Tue, Oct 5 Thu, Nov 11
Classification Graphs
6. Graph Neural Networks 1: GNN
Thu, Oct 7 Tue, Nov 16 16. Advanced Topics on GNNs
Model
7. Graph Neural Networks 2: Design
Tue, Oct 12 Thu, Nov 18 17. Scaling Up GNNs
Space
8. Applications of Graph Neural
Thu, Oct 14 Fri, Nov 19 EXAM
Networks
Tue, Oct 19 9. Theory of Graph Neural Networks Tue, Nov 30 18. Guest lecture: TBD

Thu, Oct 21 10. Knowledge Graph Embeddings Thu, Dec 2 19. GNNs for Science

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 4

  http://cs224w.stanford.edu
▪ Slides posted before the class
 Readings:
▪ Graph Representation Learning Book by
Will Hamilton
▪ Research papers
 Optional readings:
▪ Papers and pointers to additional literature
▪ This will be very useful for course projects

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 5

  Ed Discussion:
▪ Access via link on Canvas
▪ Please participate and help each other!
▪ Don’t post code, annotate your questions, search for
answers before you ask
▪ We will post course announcements to Ed (make
sure you check it regularly)
 Please don’t communicate with prof/TAs via
personal emails, but always use:
▪ [email protected]

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 6

  OHs will be virtual
▪ We will have OHs every day, starting from 2nd week
of the course
▪ See http://web.stanford.edu/class/cs224w/oh.html
for Zoom links and link to QueueStatus

Mon Tue Wed Thu Fri Sat Sun

Yige Alex Giray Weihua Federico Hongyu Serina
10:00am- 3:30pm- 1:00pm- 10:00am- 3:00pm- 9:30am- 5:00pm-
12:00pm 5:30pm 3:00pm 12:00pm 5:00pm 11:30am 7:00pm

Tracey Xuan
7:00pm- 1:00pm-
9:00pm 3:00pm

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 7

 Final grade will be composed of:
▪ Homework: 25%
▪ 3 written homeworks, each worth 8.3%
▪ Coding assignments: 20%
▪ 5 coding assignments using Google Colab, each worth 4%
▪ Exam: 35%
▪ Course project: 20%
▪ Proposal: 20%; Final report: 70%; Poster: 10%
▪ Extra credit: Ed participation, PyG/GraphGym code
contribution
▪ Used if you are on the boundary between grades
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 8
  How to submit?
▪ Upload via Gradescope
▪ You will be automatically registered to Gradescope once
you officially enroll in CS224W
▪ Homeworks, Colabs (numerical answers), and
project deliverables are submitted on Gradescope
 Total of 2 Late Periods (LP) per student
▪ Max 1 LP per assignment (no LP for the final report)
▪ LP gives 4 extra days: assignments usually due on
Thursday (11:59pm) → with LP, it is due the following
Monday (11:59pm)

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 9

  Homeworks (25%, n=3)
▪ Written assignments take longer and take time
(~10-20h) – start early!
▪ A combination of data analysis, algorithm design, and
math
 Colabs (20%, n=5)
▪ We have more Colabs but they are shorter
(~3-5h); Colab 0 is not graded.
▪ Get hands-on experience coding and training GNNs;
good preparation for final projects and industry

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 10

  Single exam: Friday, Nov 19
▪ Take-home, open-book, timed
▪ Administered via Gradescope
▪ Released at 10am PT on Friday, available until 10am
PT the following day
▪ Once you open it, you will have 100 minutes to
complete the exam
▪ Content
▪ Will have written questions (similar to Homework), will
possibly have a coding section (similar to Colabs)
▪ More details to come!

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 11

  Two options
▪ (1) Default project (predefined task)
▪ (2) Custom project (open-ended)
 Logistics
▪ Groups of up to 3 students
▪ Groups of 1 or 2 are allowed; the team size will be taken
under consideration when evaluating the scope of the project.
But 3 person teams can be more efficient.
▪ Google Cloud credits
▪ We will provide $50 in Google Cloud credits to each student
▪ You can also get $300 with Google Free Trial
(https://cloud.google.com/free/docs/gcp-free-tier)
 Read: http://cs224w.stanford.edu/info.html
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 12
 Assignment Due on (11:59pm PT)

Colab 0 Not graded

Colab 1 Thu, Oct 7 (week 3)
Homework 1 Thu, Oct 14 (week 4)
Project Proposal Tue, Oct 19 (week 5)
Colab 2 Thu, Oct 21 (week 5)
Homework 2 Thu, Oct 28 (week 6)
Colab 3 Thu, Nov 4 (week 7)
Homework 3 Thu, Nov 11 (week 8)
Colab 4 Thu, Nov 18 (week 9)
EXAM Fri, Nov 19 (week 9)
Colab 5 Thu, Dec 2 (week 11)
Project Report Thu, Dec 9 (No Late Periods!)
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 13
 Make sure you read
and understand it!

 We strictly enforce the Stanford Honor Code

▪ Violations of the Honor Code include:
▪ Copying or allowing another to copy from one’s own paper
▪ Unpermitted collaboration
▪ Plagiarism
▪ Giving or receiving unpermitted aid on a take-home examination
▪ Representing as one’s own work the work of another
▪ Giving or receiving aid on an assignment under circumstances in
which a reasonable person should have known that such aid was
not permitted
▪ The standard sanction for a first offense includes a one-
quarter suspension and 40 hours of community service.
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 14
  The course is self-contained.
 No single topic is too hard by itself.
 But we will cover and touch upon many topics
and this is what makes the course hard.
▪ Good background in:
▪ Machine Learning
▪ Algorithms and graph theory
▪ Probability and statistics
▪ Programming:
▪ You should be able to write non-trivial programs (in Python)
▪ Familiarity with PyTorch is a plus
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 15
  We use PyG (PyTorch Geometric):
▪ The ultimate library for Graph Neural Networks
 We further recommend:
▪ GraphGym: Platform for designing Graph Neural
Networks.
▪ Modularized GNN implementation, simple hyperparameter
tuning, flexible user customization
▪ Both platforms are very helpful for the course project
(save your time & provide advanced GNN
functionalities)
 Other network analytics tools: SNAP.PY, NetworkX
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 16
CS224W: Machine Learning with Graphs
Jure Leskovec, Stanford University
http://cs224w.stanford.edu
 Why Graphs?
Graphs are a general
language for describing and
analyzing entities with
relations/interactions

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 18

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 19
 Graph
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 20
 Image credit: SalientNetworks

Event Graphs Computer Networks Disease Pathways

Image credit: Wikipedia Image credit: visitlondon.com

Image credit: Pinterest

Food Webs Particle Networks Underground Networks

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 21
 Image credit: Medium Image credit: Science Image credit: Lumen Learning

Social Networks Economic Networks Communication Networks

Image credit: Missoula Current News Image credit: The Conversation

Citation Networks Internet Networks of Neurons

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 22
 Image credit: Maximilian Nickel et al Image credit: ese.wustl.edu Image credit: math.hws.edu

Knowledge Graphs Regulatory Networks Scene Graphs

Image credit: ResearchGate Image credit: MDPI Image credit: Wikipedia

Code Graphs Molecules 3D Shapes

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 23
 Image credit: Maximilian Nickel et al
Main question: Image credit: ese.wustl.edu Image credit: math.hws.edu

Knowledge Graphs Regulatory Networks Scene Graphs

How do we take advantage of
relational structure for better
prediction?

Image credit: ResearchGate Image credit: MDPI Image credit: Wikipedia

Code Graphs Molecules 3D Shapes

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 26
 Complex domains have a rich relational
structure, which can be represented as a
relational graph

By explicitly modeling relationships we

achieve better performance!

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 27

 Images

Text/Speech

Modern deep learning toolbox is designed

for simple sequences & grids
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 28
 Modern
deep learning toolbox
is designed for
sequences & grids

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 29

 Not everything
can be represented as
a sequence or a grid
How can we develop neural
networks that are much more
broadly applicable?
New frontiers beyond classic neural
networks that only learn on images
and sequences
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 30
Graphs are the new frontier
of deep learning

Graphs connect things.

Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 31

9/22/2021
 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 32
9/22/2021
Networks are complex.
 Arbitrary size and complex topological
structure (i.e., no spatial locality like grids)

vs.
Text

Networks Images
 No fixed node ordering or reference point
 Often dynamic and have multimodal features
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 33
How can we develop neural networks
that are much more broadly
applicable?
Graphs are the new frontier
of deep learning
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 34
 …
z

Predictions: Node labels,

New links, Generated
Input: Network graphs and subgraphs
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 35
 (Supervised) Machine Learning Lifecycle:
This feature, that feature. Every single time!

Raw Graph Learning

Model
Data Data Algorithm

Feature Representation Downstream

Engineering prediction task
Learning --
Automatically
learn the features
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 36
 Map nodes to d-dimensional
embeddings such that similar nodes in
the network are embedded close
together
nodeLearn a neural network representation
u

Feature representation,
embedding
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 37
 We are going to cover various topics in Machine
Learning and Representation Learning for graph
structured data:
▪ Traditional methods: Graphlets, Graph Kernels
▪ Methods for node embeddings: DeepWalk, Node2Vec
▪ Graph Neural Networks: GCN, GraphSAGE, GAT,
Theory of GNNs
▪ Knowledge graphs and reasoning: TransE, BetaE
▪ Deep generative models for graphs: GraphRNN
▪ Applications to Biomedicine, Science, Industry
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 38
 Date Topic Date Topic
1. Introduction; Machine Learning 11. Reasoning over Knowledge
Tue, Sep 21 Tue, Oct 26
for Graphs Graphs
2. Traditional Methods for ML on 12. Frequent Subgraph Mining
Thu, Sep 23 Thu, Oct 28
Graphs with GNNs
13. Community Structure in
Tue, Sep 28 3. Node Embeddings Thu, Nov 4
Networks
14. Traditional Generative
Thu, Sep 30 4. Link Analysis: PageRank Tue, Nov 9
Models for Graphs
5. Label Propagation for Node 15. Deep Generative Models for
Tue, Oct 5 Thu, Nov 11
Classification Graphs
6. Graph Neural Networks 1: GNN
Thu, Oct 7 Tue, Nov 16 16. Advanced Topics on GNNs
Model
7. Graph Neural Networks 2: Design
Tue, Oct 12 Thu, Nov 18 17. Scaling Up GNNs
Space
8. Applications of Graph Neural
Thu, Oct 14 Fri, Nov 19 EXAM
Networks
Tue, Oct 19 9. Theory of Graph Neural Networks Tue, Nov 30 18. Guest lecture: TBD

Thu, Oct 21 10. Knowledge Graph Embeddings Thu, Dec 2 19. GNNs for Science

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 39

CS224W: Machine Learning with Graphs
Jure Leskovec, Stanford University
http://cs224w.stanford.edu
 Node level

Graph-level
Community
prediction,
(subgraph)
Graph
level
generation

Edge-level

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 41

  Node classification: Predict a property of a node
▪ Example: Categorize online users / items
 Link prediction: Predict whether there are missing
links between two nodes
▪ Example: Knowledge graph completion
 Graph classification: Categorize different graphs
▪ Example: Molecule property prediction
 Clustering: Detect if nodes form a community
▪ Example: Social circle detection
 Other tasks:
▪ Graph generation: Drug discovery
▪ Graph evolution: Physical simulation
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 42
  Node classification: Predict a property of a node
▪ Example: Categorize online users / items
 Link prediction: Predict whether there are missing
links between two nodes
▪ Example: Knowledge graph completion
 GraphThese GraphCategorize
classification: ML tasks lead to
different graphs
▪ Example: Molecule property
high-impact prediction
applications!
 Clustering: Detect if nodes form a community
▪ Example: Social circle detection
 Others:
▪ Graph generation: Drug discovery
▪ Graph evolution: Physical simulation
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 43
 A protein chain acquires its native 3D structure

Image credit: DeepMind

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 45

 Computationally predict a protein’s 3D structure
based solely on its amino acid sequence

Image credit: DeepMind

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 46
 Image credit: DeepMind
Image credit: SingularityHub

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 47

  Key idea: “Spatial graph”
▪ Nodes: Amino acids in a protein sequence
▪ Edges: Proximity between amino acids (residues)

Image credit: DeepMind

Spatial graph
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 48
 Users interacts with items
▪ Watch movies, buy merchandise, listen to music
▪ Nodes: Users and items
▪ Edges: User-item interactions
 Goal: Recommend items users might like
Users
Interactions

“You might also like”

Items
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 50
 Ying et al., Graph Convolutional Neural Networks for Web-Scale Recommender Systems, KDD 2018

Task: Recommend related pins to users

Task: Learn node
embeddings such that

Query pin

Predict whether two nodes in a graph are related

9/22/2021 Jure Les kovec, Stanford CS224W: Ma chine Learning with Graphs 8
 Prescribed
Many patients take multiple drugsDrug
to treat
drugs side effect
complex or co-existing diseases:
 46% of people ages 70-79 take more than 5 drugs
 Many patients take more than 20 drugs to treat
heart disease, depression, insomnia, etc.
Task: Given a pair of drugs predict
adverse side effects

30%
, 65%
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs
prob. prob. 52
 Zitnik et al., Modeling Polypharmacy Side Effects with Graph Convolutional Networks, Bioinformatics 2018

 Nodes: Drugs & Proteins Query: How likely

 Edges: Interactions will Simvastatin and
Ciprofloxacin, when
taken together,
break down muscle
tissue?

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 53

 Zitnik et al., Modeling Polypharmacy Side Effects with Graph Convolutional Networks, Bioinformatics 2018

Drug c Drug d Evidence found

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 54

  a

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 56

  Nodes: Road segments
 Edges: Connectivity between road segments
 Prediction: Time of Arrival (ETA)

Image credit: DeepMind

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 57
 Predicting Time of Arrival with Graph Neural
Networks

 Used in Google Maps

Image credit: DeepMind

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 58
  Antibiotics are small molecular graphs
▪ Nodes: Atoms
▪ Edges: Chemical bonds

Konaklieva, Monika I. "Molecular targets of β-lactam-based antimicrobials: Image credit: CNN

beyond the usual suspects." Antibiotics 3.2 (2014): 128-142.

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 60

 Stokes et al., A Deep Learning Approach to Antibiotic Discovery, Cell 2020

 A Graph Neural Network graph classification model

 Predict promising molecules from a pool of candidates

Stokes, Jonathan M., et al. "A deep learning approach to antibiotic discovery."
Cell 180.4 (2020): 688-702.

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 61

 You et al., Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation, NeurIPS 2018

Graph generation: Generating novel molecules

Use case 1: Generate novel molecules Use case 2: Optimize existing molecules to
with high Drug likeness value have desirable properties

Drug likeness
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 62
 Sanchez-Gonzalez et al., Learning to simulate complex physics with graph networks, ICML 2020

Physical simulation as a graph:

 Nodes: Particles
 Edges: Interaction between particles

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 63

 Sanchez-Gonzalez et al., Learning to simulate complex physics with graph networks, ICML 2020

A graph evolution task:

 Goal: Predict how a graph will evolve over
time

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 64

CS224W: Machine Learning with Graphs
Jure Leskovec, Stanford University
http://cs224w.stanford.edu
  Objects: nodes, vertices N
 Interactions: links, edges E
 System: network, graph G(N,E)

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 66

 Movie 1
friend
Actor 1 Actor 2 co-worker
Peter Mary
Movie 3 Actor 4
Movie 2
friend
Tom
brothers

Actor 3 Albert

Protein 1 Protein 2
Protein 5

Protein 9 |N|=4
|E|=4
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 67
  If you connect individuals that work
with each other, you will explore a
professional network Image credit: Euro Scientists

 If you connect those that have a

sexual relationship, you will be
exploring sexual networks Image credit: ResearchGate

 If you connect scientific papers

that cite each other, you will be
studying the citation network
 If you connect all papers with the same word in the title,
what will you be exploring? It is a network, nevertheless
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 68
  How to build a graph:
▪ What are nodes?
▪ What are edges?
 Choice of the proper network representation
of a given domain/problem determines our
ability to use networks successfully:
▪ In some cases, there is a unique, unambiguous
representation
▪ In other cases, the representation is by no means
unique
▪ The way you assign links will determine the nature
of the question you can study
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 69
 Undirected Directed
 Links: undirected  Links: directed
(symmetrical, reciprocal) (arcs)
L D
A B
M C
F
I
D
E
B G G
A
H
F
C

 Examples:  Examples:
▪ Collaborations ▪ Phone calls
▪ Friendship on Facebook ▪ Following on Twitter
9/22/2021 Jure Les kovec, Stanford CS224W: Ma chine Learning with Graphs 70
 A heterogeneous graph is defined as

▪ Nodes with node types

▪ Edges with relation types
▪ Node type
▪ Relation type

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 71
 Biomedical Knowledge Graphs Academic Graphs
Example node: Migraine Example node: ICML
Example edge: (fulvestrant, Treats, Breast Neoplasms) Example edge: (GraphSAGE, NeurIPS)
Example node type: Protein Example node type: Author
Example edge type (relation): Causes Example edge type (relation): pubYear

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs, http://cs224w.stanford.edu 72
 Node degree, ki: the number
Undirected

of edges adjacent to node i

A
kA = 4
1 N
Avg. degree: k = k = å ki = 2E
N i=1 N
In directed networks we define
an in-degree and out-degree.
D
B
Directed

C
The (total) degree of a node is the
G
E sum of in- and out-degrees.
kCin = 2 kCout = 1 kC = 3
A

Source: Node with kin = 0 E

k= k in = k out
Sink: Node with kout = 0 N
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 73
 Bipartite graph is a graph whose nodes can A
be divided into two disjoint sets U and V such that
B
every link connects a node in U to one in V; that is,
U and V are independent sets
C

 Examples: D
▪ Authors-to-Papers (they authored)
E
▪ Actors-to-Movies (they appeared in)
U V
▪ Users-to-Movies (they rated)
▪ Recipes-to-Ingredients (they contain)
 “Folded” networks:
▪ Author collaboration networks
▪ Movie co-rating networks
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 74
 Projection U U V Projection V

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 75

 4
4
3
3
2
2 1
1

Aij = 1 if there is a link from node i to node j

Aij = 0 otherwise

0 1 0 1 0 0 0 1
   
1 0 0 1 1 0 0 0
A= A=
0 0 0 1 0 0 0 0
   
1 0  0 0 
 1 1  1 1

Note that for a directed graph (right) the matrix is not symmetric.
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 76
 æ0 1 0 1ö N

4 ç ÷ ki = åA ij

ç 1 0 0 1÷ j =1
Undirected

Aij =
3 ç0 0 0 1÷ N

ç ÷ k j = å Aij
2 è1 1 1 0ø i=1
1
Aij = A ji N N

L = å ki = å Aij
1 1
Aii = 0 2 i=1 2 ij

0 0 0 1
  out

4 1 0 0 0
Directed

A=
0 0 0 0 N
3 
0
 k out
in
= å Aij
0 
j
2  1 1 i=1

1 N N N
L = åk in
= åk out
= å Aij
Aij ¹ A ji i=1
i
j=1
j
i, j

Aii = 0
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 77
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 78
 Most real-world networks are sparse
E << Emax (or k << N-1)

Consequence: Adjacency matrix is filled with zeros!

(Density of the matrix (E/N2): WWW=1.51x10-5, MSN IM = 2.27x10-8)
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 79
 Represent graph as a list of edges:
▪ (2, 3)
▪ (2, 4)
2
▪ (3, 2) 1 3

▪ (3, 4)
▪ (4, 5) 5 4

▪ (5, 2)
▪ (5, 1)

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 80

  Adjacency list:
▪ Easier to work with if network is
▪ Large 1
2
3
▪ Sparse
▪ Allows us to quickly retrieve all
5 4
neighbors of a given node
▪ 1:
▪ 2: 3, 4
▪ 3: 2, 4
▪ 4: 5
▪ 5: 1, 2
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 81
 Possible options:
▪ Weight (e.g., frequency of communication)
▪ Ranking (best friend, second best friend…)
▪ Type (friend, relative, co-worker)
▪ Sign: Friend vs. Foe, Trust vs. Distrust
▪ Properties depending on the structure of the rest
of the graph: Number of common friends

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 82

  Unweighted  Weighted
(undirected) (undirected)
4 4
1 1

2 2
3 3
0 1 1 0  0 2 0.5 0
   
1 0 1 1 2 0 1 4
Aij =  Aij = 
1 1 0 0  0.5 1 0 0
   
0 1 0 0  0 4 0 0
Aii = 0 Aij = A ji Aii = 0 Aij = A ji
1 N 1 N 2E
E = å Aij k=
2E
 E =  nonzero( Aij ) k=
2 i, j=1 N 2 i , j =1 N
Examples: Friendship, Hyperlink Examples: Collaboration, Internet, Roads
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 83
  Self-edges (self-loops)  Multigraph
(undirected) (undirected)
4 4
1 1

2 2
3 3
1 1 1 0 0 2 1 0
   
 1 0 1 1  2 0 1 3
Aij = Aij =
1 1 0 0 1 1 0 0
   
0 1 0 1 0 3 0 0

Aii  0 Aij = A ji Aii = 0 Aij = A ji

1 N N
1 N 2E
E=  Aij +  Aii  E =  nonzero( Aij ) k=
2 i , j =1,i  j i =1 2 i , j =1 N
Examples: Proteins, Hyperlinks Examples: Communication, Collaboration
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 84
  Connected (undirected) graph:
▪ Any two vertices can be joined by a path
 A disconnected graph is made up by two or
more connected components
B B
A A
Largest Component:
Giant Component

Isolated node (node H)

C C
D F F
D

H H
G G

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 85

 The adjacency matrix of a network with several
components can be written in a block- diagonal
form, so that nonzero elements are confined to
squares, with all other elements being zero:
Disconnected
Connected

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 86

  Strongly connected directed graph
▪ has a path from each node to every other node
and vice versa (e.g., A-B path and B-A path)
 Weakly connected directed graph
▪ is connected if we disregard the edge directions
E
F
B

D C G Graph on the left is connected

but not strongly connected (e.g.,
there is no way to get from F to G by
following the edge directions).

9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 87

  Strongly connected components (SCCs) can
be identified, but not every node is part of a
nontrivial strongly connected component.
SCC B
E
A F
B

D E C
D C G
F
G SCC
SCC

In-component: nodes that can reach the SCC,

Out-component: nodes that can be reached from the SCC.
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 88
  Machine learning with Graphs
▪ Applications and use cases

 Different types of tasks:

▪ Node level
▪ Edge level
▪ Graph level

 Choice of a graph representation:

▪ Directed, undirected, bipartite, weighted,
adjacency matrix
9/22/2021 Jure Leskovec, Stanford CS224W: Machine Learning with Graphs 89