GL Applied Data Science Program

[email protected]
DEWPNZAQ4Y Network Analysis

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 1 / 25
Overview

Overview of this week / module:

Data collection and visualization for exploratory data analysis

Network analysis

Unsupervised learning - clustering

[email protected]
DEWPNZAQ4Y

Overview of this lecture:

Examples of networks and representing networks

Summary statistics of a network

Centrality measures - finding important nodes in a network

This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 2 / 25
Network

A network (or graph) G is a collection of nodes (or vertices) V

connected by links (or edges) E . The network is denoted by G = (V , E ).

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 3 / 25
Network

A network (or graph) G is a collection of nodes (or vertices) V

connected by links (or edges) E . The network is denoted by G = (V , E ).

Network research:
In recent years network research witnessed a big change:
[email protected]
DEWPNZAQ4Y
From study of a single graph on 10-100 nodes to the statistical
properties of large networks on millions of nodes
Characterize the structure of networks
Identify important nodes / edges in a network
Identify missing links in a network

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 3 / 25
Examples of networks

Network Vertex Edge

World Wide Web web page hyperlink
Internet computer network protocol
interaction
power grid generating station transmission line
[email protected] / substation
DEWPNZAQ4Y
friendship network person friendship
gene regulatory network gene regulatory effect
neural network neuron synapse
transportation airport direct flight
Netflix person / movie rating
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 4 / 25
Different kinds of networks
simple network: undirected network with at most one edge between
any pair of vertices and no self-loops
e.g. Internet, power grid, telephone network

multigraph: self-loops and multiple links between vertices possible

e.g. neural network, road network

directed network: i → j does not imply j → i

[email protected]
DEWPNZAQ4Y
e.g. World Wide Web, food web, citation network

weighted network: with edge weights or vertex attributes

e.g. transportation networks

bipartite network: edges between but not within classes

e.g. recommender systems such as Netflix

hypergraph: generalized ‘edges’ for interaction between > 2 nodes

This file is meant for personal use by [email protected] only.
e.g. protein-protein interaction network
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 5 / 25
Large networks look like hairballs

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 6 / 25
Representation of a network

Two common representations of a network G = (V , E ):

adjacency list
undirected graph 1 − 2 − 3: E = {{1, 2}, {2, 3}}
directed graph 1 → 2 ← 3: E = {(1, 2), (3, 2)}

[email protected]
adjacency matrix of size n × n (where n = |V |) with
DEWPNZAQ4Y

1 if (i, j) ∈ E
Aij =
0 otherwise

For weighted graph, Aij can be non-binary

How does the adjacency matrix of an undirected graph look like? How to
count
Thisthe
filenumber
is meantoffor
friends or suggest
personal new friends in a social network?
use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 7 / 25
Representation of a network

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 8 / 25
Quantitative measures of networks

Some quantitative measures of networks to describe structural patterns of

a network and to compare networks:

connected components
[email protected]
DEWPNZAQ4Y
degree distribution

diameter and average path length

homophily or assortative mixing

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 9 / 25
Connected Components
Connected component: set of nodes that are reachable from one another

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 10 / 25
Connected Components
Connected component: set of nodes that are reachable from one another
5
Many networks consist of one large component and many small ones

1e+06
Number of components
1e+04

[email protected]
DEWPNZAQ4Y
1e+02 1e+00

1e+00 1e+02 1e+04 1e+06 1e+08

Component size

Figure 3. Component size distribution. The fraction of components with a given component size
Component
onThis
a log-log issize
filescale.
meantdistribution
for personal
Most vertices
in the
use
(99.91%) are
2011 Facebook
bylargest
in the
network on a log-log
[email protected]
component. only.
scale. Most vertices (99.91%) are in the largest component.
Sharing or publishing the contents in part or full is liable for legal action.
Caroline[48].
algorithm UhlerWe
(MIT) Network
find that the average distance Analysis pairs of users was 4.7 for Great
between Learning
Facebook 10 / 25
users and
Degree distribution of the Internet
Degree of a node: number of edges connected to a node

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 11 / 25
Degree distribution of the Internet
Degree of a node: number of edges connected to a node
Many networks show a power-law degree distribution (i.e., distribution
that is linear in log-log plot)

[email protected]
DEWPNZAQ4Y

Figures from Chapter 8 in “Networks: An Introduction” by

This Newman
M.E.J. file is meant for personal use by [email protected] only.
(2010)
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 11 / 25
Degree distribution of Facebook network

1e−01

1e−04 1e−03 1e−02 1e−01 1e+00

1e−03
Fraction

Fraction
[email protected]
1e−05

DEWPNZAQ4Y

Global Glo
1e−07

U.S. U.S

1 5 50 500 5000 1 5
Degree
(a)
This file is meant for personal
From “The Anatomy ofuse by [email protected]
the Facebook only.
Social Graph” by Ugander et al. (2011)
Sharing or publishing the contents in part or full is liable for legal action.
Figure
Caroline Uhler (MIT) 1. Degree Network
distribution
Analysis p . (a) The fraction of users with
Great Learning 12 / 25de
Diameter of a graph

Let dij denote the length of the geodesic path (or shortest path)
between node i and j

The diameter of a network is the largest distance between any two

nodes in the network:
[email protected] diameter = max dij
DEWPNZAQ4Y i,j∈V

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 13 / 25
Diameter of a graph

Let dij denote the length of the geodesic path (or shortest path)
between node i and j

The diameter of a network is the largest distance between any two

nodes in the network:
[email protected] diameter = max dij
DEWPNZAQ4Y i,j∈V

If network is not connected, one often computes the diameter in the

largest component.

Algorithms for finding shortest paths: breadth-first search for

unweighted graph, Dijkstra’s algorithm for weighted graphs

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 13 / 25
Small-world and 6 degrees of separation

Concept of 6 degrees of separation was made famous by sociologist

Stanley Milgram and his study “The Small World Problem” (1967)

In his experiment participants from a particular town were asked to

get a letter to a particular person in a different town by passing it
[email protected]
from acquaintance to acquaintance.
DEWPNZAQ4Y
18 out of 96 letters made it in an average of 5.9 steps, suggesting
that the diameter of the social network in the US is 6

Any reasons why we should take the conclusion of 6 degrees of

separation with a grain of salt?

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 14 / 25
 Diameter of Facebook (2011)

Percentage of pairs within hop distance

1.0
Global
U.S.
0.6 0.8

[email protected]
0.4

DEWPNZAQ4Y
0.0 0.2

0 2 4 6 8 10
hop distance

This file is The

e 2. Diameter. meant for personal use by [email protected] only.
From “The Anatomy of the Facebook Social Graph” by Ugander et al. (2011)
neighborhood function N (h) showing the percentage of user pairs tha
Sharing
h hops of each or publishing
other. the contents
The average in part orusers
distance between full on
is liable for legal
Facebook action.
in May 2011 was 4
Caroline Uhler (MIT) Network Analysis Great Learning 15 / 25
Homophily
Homophily (or assortative mixing): tendency of people to associate with
others that are similar

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

From “The Anatomy of the Facebook Social Graph” by Ugander et al. (2011)
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 16 / 25
Find important nodes in a network
Centrality measure: A measure that captures importance of a node’s
position in the network; there are many different centrality measures:

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 17 / 25
Find important nodes in a network
Centrality measure: A measure that captures importance of a node’s
position in the network; there are many different centrality measures:
degree centrality
Simple and intuitive: individuals with more connections have more
influence and more access to information.
Does not capture “cascade of effects”: importance better captured by
having connections to important nodes
[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 17 / 25
Find important nodes in a network
Centrality measure: A measure that captures importance of a node’s
position in the network; there are many different centrality measures:
degree centrality
Simple and intuitive: individuals with more connections have more
influence and more access to information.
Does not capture “cascade of effects”: importance better captured by
having connections to important nodes
[email protected]
DEWPNZAQ4Y
eigenvector centrality
score that is proportional to the sum of the score of all neighbors is
captured by largest eigenvector of adjacency matrix
builds the foundation for Google’s PageRank algorithm

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 17 / 25
Find important nodes in a network
Centrality measure: A measure that captures importance of a node’s
position in the network; there are many different centrality measures:
degree centrality
Simple and intuitive: individuals with more connections have more
influence and more access to information.
Does not capture “cascade of effects”: importance better captured by
having connections to important nodes
[email protected]
DEWPNZAQ4Y
eigenvector centrality
score that is proportional to the sum of the score of all neighbors is
captured by largest eigenvector of adjacency matrix
builds the foundation for Google’s PageRank algorithm
closeness centrality
tracks how close a node is to any other node

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 17 / 25
Find important nodes in a network
Centrality measure: A measure that captures importance of a node’s
position in the network; there are many different centrality measures:
degree centrality
Simple and intuitive: individuals with more connections have more
influence and more access to information.
Does not capture “cascade of effects”: importance better captured by
having connections to important nodes
[email protected]
DEWPNZAQ4Y
eigenvector centrality
score that is proportional to the sum of the score of all neighbors is
captured by largest eigenvector of adjacency matrix
builds the foundation for Google’s PageRank algorithm
closeness centrality
tracks how close a node is to any other node
betweenness centrality
This file is meant for personal use by [email protected] only.
measures the extent to which a node lies on paths between other nodes
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 17 / 25
Which centrality measure to use

Choice of centrality measure depends on application!

[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 18 / 25
Which centrality measure to use

Choice of centrality measure depends on application!

In a friendship network:

high degree centrality: most popular person

[email protected]
DEWPNZAQ4Y
high eigenvector centrality: most popular person that is friends with
popular people

high closeness centrality: person that could best inform the group

high betweenness centrality: person whose removal could best break

the network apart

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 18 / 25
Case study: CAVIAR (criminal network in Montreal)

Data based on 11 wiretap warrants from 1994-1996 → 11 periods

Mandate of CAVIAR project: Seize drugs, arrests only in period 11

11 seizures total with monetary losses for traffickers of $32 mio
phase 4: 1 seizure $ 2.5mio, 300kg of marijuana
phase 6: 3 seizures $ 1.3mio, 2 x 15kg of marijuana, 1 x 2 kg of
[email protected]
cocaine
DEWPNZAQ4Y
phase 7: 1 seizure $ 3.5mio, 401kg of marijuana
phase 8: 1 seizure $ 0.4mio, 9kg of cocaine
phase 9: 2 seizures $ 4.3mio, 2kg of cocaine + 1 x 500kg marijuana
phase 10: 1 seizure $ 18.7mio, 2200kg of marijuana
phase 11: 2 seizures $ 1.3mio, 12kg of cocaine + 11kg of cocaine

Unique opportunity to study changes in the structure of a criminal network

This file isby
in upheaval meant
policeforforces
personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 19 / 25
Case study: CAVIAR (criminal network in Montreal)
network consists of 110 (numbered) players: 1-82 are traffickers,
83-110 are non-traffickers (financial investors, accountants, owners of
various importation businesses, etc.)
initially, investigation targeted Daniel Serero, alleged mastermind of
drug network in downtown Montreal
initially marijuana
Project was imported to Canada from Morocco
Caviar
[email protected]
after first (The
DEWPNZAQ4Y seizurepolice bias
in phase 4, and the scope
traffickers of law-
reoriented to cocaine import
enforcement
from Colombia, transitingmonitoring)
through the United States
45
40
35
# de participants

30
25
20
15
10
5
0
1 2 3 4 5 6 7 8 9 10 11

This file is meant for personal use byPhase

[email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 20 / 25
Case study: CAVIAR (criminal network in Montreal)

Phase 3
Phase 6

[email protected]
DEWPNZAQ4Y
Phase 8 Phase 11

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 21 / 25
Case study: CAVIAR (criminal network in Montreal)
Role of the different actors:
Daniel Serero (node 1): mastermind of the network
Pierre Perlini (node 3): principal lieutenant of Serero (executes his
instructions)
Ernesto Morales (node 12): principal organizer of the cocaine import,
intermediary between the Colombians and the Serero organization
[email protected]
DEWPNZAQ4Y

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 22 / 25
Optional: Additional thoughts - Criminal networks

Given a social network and k criminal suspects, how to determine

other suspects?

Same question is extremely important in biology: given certain genes

that are known to cause a certain disease, determine other candidate
[email protected]
DEWPNZAQ4Y
genes (e.g. based on protein-protein interaction network for
determining autism genes: http://dx.doi.org/10.1101/057828)

How do we identify nodes that are “between” a given set of seed

nodes?

This file is meant for personal use by [email protected] only.

Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 23 / 25
Optional: Steiner trees
Determine a small subnetwork that contains the given suspects / genes
and connects these nodes

Steiner tree:
shortest subnetwork that contains a given set of nodes
NP-complete problem
[email protected]
there exist polynomial time approximations
DEWPNZAQ4Y

⇒ use collection of approximate Steiner trees for further analysis:

autism interactome / criminal interactome
For genomics applications, see:
http://fraenkel-nsf.csbi.mit.edu/steinernet/tutorial.html

⇒ compute nodes with high betweenness centrality in interactome to

This file is meant for personal use by [email protected] only.
obtain candidate genes / suspects
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 24 / 25
References

Chapters 1 - 10 (but mostly chapters 6 - 8) in

M. E. J. Newman. Networks: An Introduction. 2010.

For an analysis of the Facebook network:

[email protected]
J. Ugander, B. Karrer, L. Backstrom and C. Marlow. The Anatomy of
DEWPNZAQ4Y
the Facebook Social Graph. 2011.

For more information on the CAVIAR network:

C. Morselli. Inside Criminal Networks (Springer, New York).
Chapter 6: Law-enforcement disruption of a drug-importation
network. 2009.
This file is meant for personal use by [email protected] only.
Sharing or publishing the contents in part or full is liable for legal action.
Caroline Uhler (MIT) Network Analysis Great Learning 25 / 25