SlideShare a Scribd company logo

Streaming a Million Likes/Second: Real-Time Interactions on Live Video

ā€¢
C4Media, profile picture
C4Media

The document discusses real-time interactions during live video streaming, highlighting infrastructure challenges and architectural strategies to support high engagement metrics like likes and comments. It emphasizes the use of Akka and distributed systems to manage connections and ensure reliability for real-time delivery, aiming to scale to thousands of concurrent viewers. Additionally, it outlines subscription flows and event publishing mechanisms to handle large volumes of interactions efficiently.

Technology
Related topics:
architecture-design ā€¢ Distributed Systems Overview
1 of 171
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
Streaming a million likes/second Real-time interactions on Live Video Akhilesh Gupta Realtime Engineer, LinkedIn 2
InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ linkedin-play-akka-distributed- systems/ ā€¢ Over 1,000,000 software developers, architects and CTOs read the site world- wide every month ā€¢ 250,000 senior developers subscribe to our weekly newsletter ā€¢ Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) ā€¢ Post content from our QCon conferences ā€¢ 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building ā€¢ 96 deep dives on innovative topics packed as downloadable emags and minibooks ā€¢ Over 40 new content items per week
Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
tiny.cc/qconlive 3 Login with LinkedIn credentials
Which was the largest live stream in the world? (largest number of concurrent viewers) ? 4
INDIA VS NEW ZEALAND WORLD CUP Largest Live Stream >25.3M viewers Ā© The Indian Express https://images.indianexpress.com/2019/06/13th-june_india-vs-new-zealand_759.jpg 5
6
D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 7
8
What is a Live Video? 9
What are real-time interactions on Live Video? 10
Mobile LINKEDIN LIVE Likes/Comments Concurrent Viewer Counts 11
LINKEDIN LIVE Desktop Likes/Comments Concurrent Viewer Counts 12
Distribution of Likes A S 13
Distribution of Likes A S Simple HTTP Request 14
Distribution of Likes A S Simple HTTP Request ? 15
Challenge 1:The delivery pipe 16
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) 17
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Simple HTTP Request 18
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 19
The delivery pipe Real-time Delivery A S Likes Backend (Publisher) Persistent Connection 20
Persistent Connection Real-time Delivery 21
TRY IT! Login at linkedin.com Persistent Connection tiny.cc/realtime 22
23
EVENTSOURCE INTERFACE CLIENT REQUEST HTTP Long Poll with Server Sent Events GET /connect HTTP/1.1 Accept: text/event-stream 24
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream 25
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {ā€œlikeā€ object} 26
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {ā€œlikeā€ object} .. data: {ā€œcommentā€ object} 27
EVENTSOURCE INTERFACE SERVER RESPONSE HTTP Long Poll with Server Sent Events HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {ā€œlikeā€ object} .. data: {ā€œcommentā€ object} .. data: {ā€œlikeā€ object} 28
WEB EventSource Client Libraries var evtSource = new EventSource("https://www.linkedin.com/realtime/connect"); evtSource.onmessage = function(e) { var likeObject = JSON.parse(e.data); } 29
ANDROID & IOS SUPPORT EventSource Client Libraries Android: https://github.com/tylerjroach/eventsource-android iOS Swift: https://github.com/inaka/EventSource 30
Next Challenge? Real-time Delivery 31
Next Challenge: Multiple Connections Real-time Delivery 32
Challenge 2: Connection Management 33
PLAY & AKKA Connection Management Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications 34
Ā© Disney, All Rights Reserved, Pirates of the Caribbean 35
AN AKKA ACTOR Connection Management State Behavior 36
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor 37
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor 38
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor Millions of Actors 39
AN AKKA ACTOR Connection Management State Behavior ABC Mailbox One per Actor No. of threads proportional to no. of cores Millions of Actors 40
AN AKKA ACTOR Connection Management Connection Publish Event ABC Events to be published Millions of Connections 41
INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { } 42
INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { return ok(EventSource.whenConnected(eventSource -> { String connectionId = UUID.randomUUID().toString(); })); } 43
INITIALIZE AN AKKA ACTOR Server: Accepting a EventSource connection // Client A connects to the server and is assigned connectionIdA public Result listen() { return ok(EventSource.whenConnected(eventSource -> { String connectionId = UUID.randomUUID().toString(); // construct an Akka Actor to manage connection _actorSystem.actorOf( ClientConnectionActor.props(connectionId, eventSource), connectionId ); })); } 44
Publish Events Using Akka Actors Akka Actor Real-time Delivery System Connection ID cid1 45
EVENT PUBLISH FROM PUBLISHER Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 46
EVENT PUBLISH FROM SUPERVISOR TO CHILD ACTORS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 47
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 48
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publishing on EventSource eventSource.send(<like object>); 49
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 50
EVENT PUBLISH FROM CHILD ACTORS TO CLIENT CONNECTIONS Publish Events Using Akka Actors Actor 1 Real-time Delivery System Actor 2 Actor 3 Actor 4 Actor 5 Akka Supervisor Actor cid1 cid2 cid3 cid4 cid5 51
EVENTSOURCE INTERFACE Received Events on the Clients HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream - RESPONSE BODY - data: {ā€œUser A liked the videoā€¦ā€ object} 52
Next Challenge? Real-time Delivery 53
Next Challenge: Multiple Live Videos Real-time Delivery 54
Challenge 3:Multiple Live Videos 55
START WATCHING LIVE VIDEO Multiple Live Videos cid3 cid5 Live Video 1 Live Video 2 Real-time Delivery System 56
HTTP SUBSCRIPTION REQUEST Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 Live Video 1 Live Video 2 /subscribe 57
IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid5 Live Video 2 /subscribe /subscribe cid3Live Video 1 58
IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribe to Live Videos Topic ConnectionId In-Memory Subscriptions Table cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe 59
Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 60
PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 61
SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 62
PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 63
PUBLISHER PUBLISH TO SUPERVISOR ACTOR Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 64
SUBSCRIBER LOOKUP FROM IN MEMORY TABLE Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 65
PUBLISH TO CLIENT CONNECTIONS Publish using Subscription Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 66
Next Challenge? Real-time Delivery 67
Next Challenge: 10K Concurrent Viewers Real-time Delivery 68
Challenge 4:10K Concurrent Viewers 69
70
10K Concurrent Viewers Real-time Delivery 71
S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 72
10K Concurrent Viewers Real-time Delivery 73
10K Concurrent Viewers Frontend Frontend 74
10K Concurrent Viewers Frontend Frontend 75
DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 76
DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 77
DISPATCHES BETWEEN FRONTEND NODES Real-time Dispatcher Frontend Frontend Real-time Dispatcher 78
Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Real-time Dispatcher 79 Live Video 1
HTTP SUBSCRIPTION REQUEST Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe 80 Live Video 1
SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 81 Live Video 1
SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 82 node2 Live Video 1
SUBSCRIPTIONS TABLE ENTRY Frontend Node Subscriptions node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host Real-time Dispatcher Subscriptions Table /subscribe /subscribe 83 node2 Live Video 1
Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2 84
PUBLISHER PUBLISH TO DISPATCHER Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 85 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
SUBSCRIBER LOOKUP Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 86 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
PUBLISH TO SUBSCRIBED FRONTEND NODES Publish to Frontend Nodes node1 Real-time Dispatcher Frontend node2 Frontend node3 Frontend node4 Frontend node5 Frontend Dispatcher 87 Live Video 1 node1 Live Video 2 node5 Live Video 2 node3 Live Video 1 node2 Live Video 2 node2
Frontend Node to Client Publish Frontend 88
Frontend Node to Client Publish Frontend 89
Whatā€™s the next challenge? Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 90
Challenge 5:100 Likes/second 91
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 92
S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 93
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 94
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 95
100 Likes/second Frontend Frontend Frontend Frontend Frontend Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 96
100 Likes/second Frontend Frontend Frontend Frontend Frontend Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Dispatcher Frontend Frontend 97
COUCHBASE/REDIS ETC. Key-Value Store for Subscriptions Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 Key-Value Store Dispatcher 98
Publish to Dispatcher Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 99
Subscriber lookup from KV Store Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 100
Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 101
Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 102
Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 103
Challenge 6: 100 Likes/s, 10K Viewers Distribution of 1M Likes/s 104
Mobile LINKEDIN LIVE Likes 105
The Subscription Flow Frontend Dispatcher Key-Value Store 1 106
START WATCHING LIVE VIDEO Viewers subscribing to Live Video Likes cid3 cid5 Live Video 1 Live Video 2 Frontend 107
HTTP SUBSCRIPTION REQUEST Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe Frontend In-Memory Subscriptions Table 108
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid5 Live Video 2 /subscribe /subscribe cid3Live Video 1 Frontend In-Memory Subscriptions Table 109
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe Frontend In-Memory Subscriptions Table 110
FRONTEND IN-MEMORY SUBSCRIPTION ENTRY Viewers subscribing to Live Video Likes Topic ConnectionId cid3 cid5 Live Video 1 Live Video 2 /subscribe /subscribe Frontend In-Memory Subscriptions Table 111
The Subscription Flow Frontend Dispatcher Key-Value Store 1 2 3 112
FRONTEND RECEIVES SUBSCRIPTION REQUEST Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Key-Value Store Dispatcher Dispatcher 113
FRONTEND HTTP SUBSCRIPTION REQUEST TO DISPATCHER Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe Key-Value Store Dispatcher Dispatcher 114
DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe /subscribe Dispatcher Dispatcher Key-Value Store 115
DISPATCHER SUBSCRIPTION ENTRY IN KV STORE Frontend Node Subscriptions in Dispatcher node1 node2 Live Video 1 Live Video 2 Frontend Frontend Topic Frontend Host /subscribe /subscribe Dispatcher Dispatcher Key-Value Store 116
The Publish Flow Frontend Dispatcher Key-Value Store 1 117
Viewers liking Live Videos Likes Backend (Publisher) 118
Likes Published to Backend Likes Backend (Publisher) 119
The Publish Flow Frontend Dispatcher Key-Value Store 1 2 345 120
The Publish Flow Frontend Dispatcher Key-Value Store 2 345 121
BACKEND PUBLISH TO DISPATCHER NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node3 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 122
SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 123
SUBSCRIBER LOOKUP FROM KV STORE Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 node1 node2 node3 node4 node5 node6 node7 124
PUBLISH TO FRONTEND NODES Dispatcher Publish to Frontend Nodes Frontend Frontend Frontend Frontend Frontend Dispatcher Dispatcher Frontend Frontend Key-Value Store Dispatcher node1 node2 node3 node4 node5 node6 node7 Live Video 1 node7 Live Video 2 node5 Live Video 2 node1 Live Video 1 node4 Live Video 2 node2 125
The Publish Flow Frontend Dispatcher Key-Value Store 5 126
DISPATCHER PUBLISH TO FRONTEND NODE SUPERVISOR ACTOR Frontend Publish to Mobile Clients Actor 1 Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher Frontend In-Memory Subscriptions Table 127
SUBSCRIBED CLIENT CONNECTION LOOKUP FROM IN-MEMORY TABLE Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher In-Memory Subscriptions Table 128
PUBLISH TO MOBILE CLIENTS Frontend Publish to Mobile Clients Actor 1 Frontend Actor 2 Actor 3 Actor 4 Actor 5 Supervisor cid1 cid2 cid3 cid4 cid5 Live Video 1 cid3 Live Video 2 cid5 Live Video 2 cid1 Live Video 1 cid4 Live Video 2 cid2 Dispatcher In-Memory Subscriptions Table 129
Bonus Challenge: Another data center 130
131
D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 132
Frontend DispatcherDC-1 133
Frontend Dispatcher Frontend DispatcherDC-2 DC-1 134
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 135
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 136
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 137
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 138
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 CrossDataCenterPublish 139
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 140
Frontend Dispatcher Frontend Dispatcher Frontend Dispatcher DC-3 DC-2 DC-1 141
Performance & Scale 142
How many persistent connections can a single machine hold? Frontend Server ? 143
> 100,000 144
ROYAL WEDDING Second Largest Stream >18M viewers Ā© time.com 145
> 100,000 We can hold those 18M connections with just 180 machines 146
LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinscaling >100K 147
LinkedIn Realtime Performance Persistent Connections/machine 5K/s Dispatcher Publish QPS/machine 75ms E2E Publish Latency >100K 148
Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher How many events per second can be published to a single dispatcher machine? ? events/second 149
Dispatcher Performance Number of frontend nodes to publish to is limited per event Frontend Frontend Frontend Frontend Frontend Dispatcher ? events/second 150
Dispatcher Performance Frontend Frontend Frontend Frontend Frontend Dispatcher ? events/second 151
5K We can publish 50K likes per second with just 10 machines 152
LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency 5K 153
E2E Publish Latency Frontend Dispatcher t1 Key-Value Store 154
E2E Publish Latency Frontend Dispatcher t1 Key-Value Store t2 t2-t1 = 75ms p90 155
LinkedIn Realtime Performance >100K Persistent Connections/machine Dispatcher Publish QPS/machine 75ms E2E Publish Latency 5K 156
LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinlatency Measurement System E2E Publish Latency 157
Why does this system scale? Horizontally Scalable System Frontend Dispatcher Frontend Frontend Frontend Dispatcher Dispatcher Dispatcher Distributed K/V Storage System 158
Active Status/Presence Online/Oļ¬€line indicators LINKEDIN REALTIME TECHNOLOGY 159
LINKEDIN ENGINEERING BLOG tiny.cc/ linkedinpresence Active Status/Presence 160
Key takeaways 161
R E A LT I M E I N T E R A C T I O N S Enable dynamic instant experiences on your apps 162
E V E N T S O U R C E / S S E Built-in support in most server frameworks Readily available client libraries HTTP/1.1 200 OK - RESPONSE HEADERS - Content-Type: text/event-stream 163
P l a y F r a m e w o r k a n d A k k a A c t o r s Powerful frameworks to manage millions of connections State BehaviorMailbox 164
D i s t r i b u t e d S y s t e m s Start small and add simple layers to your architecture 165
S C A L I N G B A C K E N D S Y S T E M S When in doubt, add a machine! 166
R E A LT I M E I N T E R A C T I O N P L A T F O R M Can be built on any server/storage technology Horizontally scalable Frontend Dispatcher Key- 167
R E A LT I M E I N T E R A C T I O N P L A T F O R M You can do it for your app! 168
R E A LT I M E E N G I N E E R , L I N K E D I N @agupta03 t i n y . c c / q c o n 2 0 2 0 169 AMA at 1.40pm at Guild, Mezzanine
Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ linkedin-play-akka-distributed-systems/

More Related Content

PPTX
Pubg topic
Darshan Reddy
Ā 
PPTX
Pubg
Arpit Sem
Ā 
PDF
Generation of Computers
rehnamca
Ā 
PPTX
LiveOps 101 | Andrew Munden
Jessica Tams
Ā 
PDF
Game Design - Lecture 1
Andrea Resmini
Ā 
PDF
LiveOps as a Service | Scott Humphries
Jessica Tams
Ā 
PPT
Endless runner
PratikPatil327
Ā 
PDF
Effective LiveOps Strategies for F2P Games
James Gwertzman
Ā 
Pubg topic
Darshan Reddy
Ā 
Pubg
Arpit Sem
Ā 
Generation of Computers
rehnamca
Ā 
LiveOps 101 | Andrew Munden
Jessica Tams
Ā 
Game Design - Lecture 1
Andrea Resmini
Ā 
LiveOps as a Service | Scott Humphries
Jessica Tams
Ā 
Endless runner
PratikPatil327
Ā 
Effective LiveOps Strategies for F2P Games
James Gwertzman
Ā 

Similar to Streaming a Million Likes/Second: Real-Time Interactions on Live Video (20)

PDF
QCon London 2020: Streaming a million likes/second Real-time interactions on...
Akhilesh Gupta
Ā 
PDF
Codemotion 2019: A million likes/second: Real-time interactions on Live Video
Akhilesh Gupta
Ā 
PDF
Realtime Content Delivery: Powering dynamic instant experiences
Akhilesh Gupta
Ā 
PDF
Springone2gx 2014 Reactive Streams and Reactor
StƩphane Maldini
Ā 
PDF
Akka Streams and HTTP
Roland Kuhn
Ā 
PDF
How Netflix Directs 1/3rd of Internet Traffic
C4Media
Ā 
PDF
ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
DATAVERSITY
Ā 
PDF
Information sharing pipeline
Violeta Ilik
Ā 
PDF
Over the Top Content Delivery: State of the Art and Challenges Ahead
Alpen-Adria-UniversitƤt
Ā 
PPTX
A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
Legacy Typesafe (now Lightbend)
Ā 
PDF
NRT Event Processing with Guaranteed Delivery of HTTP Callbacks, HBaseCon 2015
Cask Data
Ā 
PDF
HBaseCon 2015: NRT Event Processing with Guaranteed Delivery of HTTP Callbacks
HBaseCon
Ā 
PDF
DevOps in the Amazon Cloud ā€“ Learn from the pioneersNetflix suro
Gaurav "GP" Pal
Ā 
PDF
Engage 2020: Hello are you listening, There is stream for everything
Frank van der Linden
Ā 
PDF
Reactive Stream Processing with Akka Streams
Konrad Malawski
Ā 
PPTX
Streaming Stored Video
MdAshikJiddney
Ā 
PDF
Reactive Streams 1.0.0 and Why You Should Care (webinar)
Legacy Typesafe (now Lightbend)
Ā 
PDF
Event Driven-Architecture from a Scalability perspective
Jonas BonƩr
Ā 
KEY
Event Driven Architecture
andreaskallberg
Ā 
PDF
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
LinkedIn
Ā 
QCon London 2020: Streaming a million likes/second Real-time interactions on...
Akhilesh Gupta
Ā 
Codemotion 2019: A million likes/second: Real-time interactions on Live Video
Akhilesh Gupta
Ā 
Realtime Content Delivery: Powering dynamic instant experiences
Akhilesh Gupta
Ā 
Springone2gx 2014 Reactive Streams and Reactor
StƩphane Maldini
Ā 
Akka Streams and HTTP
Roland Kuhn
Ā 
How Netflix Directs 1/3rd of Internet Traffic
C4Media
Ā 
ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
DATAVERSITY
Ā 
Information sharing pipeline
Violeta Ilik
Ā 
Over the Top Content Delivery: State of the Art and Challenges Ahead
Alpen-Adria-UniversitƤt
Ā 
A Deeper Look Into Reactive Streams with Akka Streams 1.0 and Slick 3.0
Legacy Typesafe (now Lightbend)
Ā 
NRT Event Processing with Guaranteed Delivery of HTTP Callbacks, HBaseCon 2015
Cask Data
Ā 
HBaseCon 2015: NRT Event Processing with Guaranteed Delivery of HTTP Callbacks
HBaseCon
Ā 
DevOps in the Amazon Cloud ā€“ Learn from the pioneersNetflix suro
Gaurav "GP" Pal
Ā 
Engage 2020: Hello are you listening, There is stream for everything
Frank van der Linden
Ā 
Reactive Stream Processing with Akka Streams
Konrad Malawski
Ā 
Streaming Stored Video
MdAshikJiddney
Ā 
Reactive Streams 1.0.0 and Why You Should Care (webinar)
Legacy Typesafe (now Lightbend)
Ā 
Event Driven-Architecture from a Scalability perspective
Jonas BonƩr
Ā 
Event Driven Architecture
andreaskallberg
Ā 
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
LinkedIn
Ā 
Ad

More from C4Media (20)

PDF
Next Generation Client APIs in Envoy Mobile
C4Media
Ā 
PDF
Software Teams and Teamwork Trends Report Q1 2020
C4Media
Ā 
PDF
Understand the Trade-offs Using Compilers for Java Applications
C4Media
Ā 
PDF
Kafka Needs No Keeper
C4Media
Ā 
PDF
High Performing Teams Act Like Owners
C4Media
Ā 
PDF
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
C4Media
Ā 
PDF
Service Meshes- The Ultimate Guide
C4Media
Ā 
PDF
Shifting Left with Cloud Native CI/CD
C4Media
Ā 
PDF
CI/CD for Machine Learning
C4Media
Ā 
PDF
Fault Tolerance at Speed
C4Media
Ā 
PDF
Architectures That Scale Deep - Regaining Control in Deep Systems
C4Media
Ā 
PDF
ML in the Browser: Interactive Experiences with Tensorflow.js
C4Media
Ā 
PDF
Build Your Own WebAssembly Compiler
C4Media
Ā 
PDF
User & Device Identity for Microservices @ Netflix Scale
C4Media
Ā 
PDF
Scaling Patterns for Netflix's Edge
C4Media
Ā 
PDF
Make Your Electron App Feel at Home Everywhere
C4Media
Ā 
PDF
The Talk You've Been Await-ing For
C4Media
Ā 
PDF
Future of Data Engineering
C4Media
Ā 
PDF
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
C4Media
Ā 
PDF
Navigating Complexity: High-performance Delivery and Discovery Teams
C4Media
Ā 
Next Generation Client APIs in Envoy Mobile
C4Media
Ā 
Software Teams and Teamwork Trends Report Q1 2020
C4Media
Ā 
Understand the Trade-offs Using Compilers for Java Applications
C4Media
Ā 
Kafka Needs No Keeper
C4Media
Ā 
High Performing Teams Act Like Owners
C4Media
Ā 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
C4Media
Ā 
Service Meshes- The Ultimate Guide
C4Media
Ā 
Shifting Left with Cloud Native CI/CD
C4Media
Ā 
CI/CD for Machine Learning
C4Media
Ā 
Fault Tolerance at Speed
C4Media
Ā 
Architectures That Scale Deep - Regaining Control in Deep Systems
C4Media
Ā 
ML in the Browser: Interactive Experiences with Tensorflow.js
C4Media
Ā 
Build Your Own WebAssembly Compiler
C4Media
Ā 
User & Device Identity for Microservices @ Netflix Scale
C4Media
Ā 
Scaling Patterns for Netflix's Edge
C4Media
Ā 
Make Your Electron App Feel at Home Everywhere
C4Media
Ā 
The Talk You've Been Await-ing For
C4Media
Ā 
Future of Data Engineering
C4Media
Ā 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
C4Media
Ā 
Navigating Complexity: High-performance Delivery and Discovery Teams
C4Media
Ā 
Ad

Recently uploaded (20)

PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
Ā 
PDF
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
Ā 
PPTX
Smart Infrastructure and Automation through IoT Sensors
Rejig Digital
Ā 
PDF
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
Ā 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
Ā 
PDF
Revolutionize Operations with Intelligent IoT Monitoring and Control
Rejig Digital
Ā 
PDF
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
Ā 
PDF
DevOps & Developer Experience Summer BBQ
AUGNYC
Ā 
PDF
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
Ā 
DOCX
Top AI API Alternatives to OpenAI: A Side-by-Side Breakdown
vilush
Ā 
PDF
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
Ā 
PDF
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
Ā 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
Ā 
PDF
Enable Enterprise-Ready Security on IBM i Systems.pdf
Precisely
Ā 
PDF
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
Ā 
PDF
Software Development Methodologies in 2025
KodekX
Ā 
PDF
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
Ā 
PDF
The Evolution of KM Roles (Presented at Knowledge Summit Dublin 2025)
Enterprise Knowledge
Ā 
PDF
Orbitly Pitch Deckļ½œA Mission-Driven Platform for Side Project Collaboration (...
zz41354899
Ā 
PDF
Doc9.....................................
SofiaCollazos
Ā 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
Ā 
solutions_manual_-_materials___processing_in_manufacturing__demargo_.pdf
AbdullahSani29
Ā 
Smart Infrastructure and Automation through IoT Sensors
Rejig Digital
Ā 
Google I/O Extended 2025 Baku - all ppts
HusseinMalikMammadli
Ā 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
Ā 
Revolutionize Operations with Intelligent IoT Monitoring and Control
Rejig Digital
Ā 
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
Ā 
DevOps & Developer Experience Summer BBQ
AUGNYC
Ā 
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
Ā 
Top AI API Alternatives to OpenAI: A Side-by-Side Breakdown
vilush
Ā 
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
Ā 
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
Ā 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
Ā 
Enable Enterprise-Ready Security on IBM i Systems.pdf
Precisely
Ā 
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
Ā 
Software Development Methodologies in 2025
KodekX
Ā 
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
Ā 
The Evolution of KM Roles (Presented at Knowledge Summit Dublin 2025)
Enterprise Knowledge
Ā 
Orbitly Pitch Deckļ½œA Mission-Driven Platform for Side Project Collaboration (...
zz41354899
Ā 
Doc9.....................................
SofiaCollazos
Ā 

Streaming a Million Likes/Second: Real-Time Interactions on Live Video