A 7 architecture sustaining line live
14 likes•13,647 views
LINE LIVE is a video streaming and hosting service that allows users to watch streams from celebrities and companies or host their own streams. It uses HTTP Live Streaming (HLS) to deliver video content through media servers to viewers via a CDN. Streams can be viewed on LINE and chat functionality allows users to communicate and show appreciation through "Loves". The service has grown significantly since its launch in 2014 and now handles over 10,000 chat messages and millions of "Loves" per stream through scaled infrastructure including servers, storage systems and a Redis cluster to support low-latency interactions.
A 7 architecture sustaining line live
- 3. What is LINE LIVE? Video streaming and hosting service Watch streams from celebrities and companies Host or watch streams from other users Chat communication Showing appreciation with “Loves”
- 4. History of LINE LIVE Feb 2014 LINE LIVE CAST was released
- 5. History of LINE LIVE Feb 2014 LINE LIVE CAST was released Sep 2015 Kick off meeting for LINE LIVE
- 6. History of LINE LIVE Feb 2014 LINE LIVE CAST was released Sep 2015 Kick off meeting for LINE LIVE Dec 2015 LINE LIVE was released
- 7. History of LINE LIVE Feb 2014 LINE LIVE CAST was released Sep 2015 Kick off meeting for LINE LIVE Dec 2015 LINE LIVE was released Aug 2016 LINE LIVE Version 2 is out
- 8. Tools and Staff Java, Nginx, MySQL, Redis, Elasticsearch, Fluentd, CDN, etc. Server (media 260+, chat 110+, API and other 100+) Over 20 core engineers Over 30 other related engineers (LINE app, storage, network, etc.)
- 9. LINE LIVE Architecture OBS Inhouse Storage System Hosting Viewing RTMP Media Servers API Servers LINE Billing CMS Chat WebSocket JSON API CDN Akamai LINE ServersUpload HLS Files Cache Send Operation On JSON API Send Events On JSON API
- 11. Data Analysis For details, see the presentation, “New Stream Processing Platform with Apache Flink" Data Labs 荻林裕憲 Real-time viewer aggregation Streaming program KPI measurement, etc.
- 15. HLS All compiled into one video live-1_001.ts live-1_002.ts live-1_003.ts live-1_004.ts live-1_005.ts (HTTP Live Streaming)
- 20. Architecture Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing RTMP
- 21. Architecture Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing RTMP Upload HLS Files
- 22. Architecture Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing Send Events On JSON API RTMP Upload HLS Files
- 23. Architecture Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing Send Events On JSON API RTMP Send a Push Upload HLS Files
- 24. Architecture Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing Send Events On JSON API Cache Download HLS Files RTMP JSON API Upload HLS Files
- 25. HLS Good Hack on LINE LIVE
- 26. HLS Good Hack for Data LIVE http://〜/{encrypt(objectId, startAt, durationSec)}_720_xxx.ts http://〜/{encrypt(objectId, startAt, durationSec)}.m3u8 http://〜/FooBar_720_xxx.ts http://〜/FooBar.m3u8 Example : startAt=19:00 durationSec=3600 Prepare encrypted URL on server in advance
- 27. HLS Good Hack for Data LIVE GET 200 GET 200 18:00 21:0019:00 20:00 http://〜/FooBar.m3u8 http://〜/FooBar_720_xxx.ts
- 28. HLS Good Hack for Data LIVE Example : startAt=19:00 durationSec=3600 GET 404 GET 404 GET 404 GET 404 18:00 21:0019:00 20:00 http://〜/FooBar.m3u8 http://〜/FooBar_720_xxx.ts
- 29. HLS Good Hack for Auto play http://〜/{encrypt(objectId, startAt, durationSec)}.m3u8 http://〜/FooBar_720_xxx.ts Prepare encrypted URL on server in advance Example : startAt=2sec durationSec=6 001.ts 002.ts 003.ts 004.ts 005.ts xxx.ts …
- 30. HLS Good Hack for Auto play Example : startAt=2sec durationSec=6 001.ts 002.ts 003.ts 004.ts 005.ts xxx.ts … GET 200 GET GET GET http://〜/AcmeAcme.m3u8 foo-bar.m3u8 ・002.ts ・003.ts ・004.ts
- 31. Flow When Stream Is Terminated Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing RTMP Upload HLS Files
- 32. Stream Terminated Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing RTMP Upload HLS Files
- 33. Stream Terminated Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing 2. Cache RTMP Upload HLS Files 1. GET HLS2. Fetch Origin
- 34. Stream Terminated Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing 3. 404 4. 404 RTMP Upload HLS Files 1. GET HLS2. Fetch Origin
- 35. Standby Until Recovery Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing 3. 404 4. 404 RTMP Upload HLS Files Burst API Server 6. SUSPEND 5. Polling SUSPEND Event 1. GET HLS2. Fetch Origin
- 36. Recovery Flow Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing 1. RTMP restart Upload HLS Files Burst API Server 2. RESTART event
- 37. Recovery Flow Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing 1. RTMP restart Upload HLS Files Burst API Server 4. RESTART 3. Polling 2. RESTART event
- 38. Recovery Flow Media Servers API Servers OBS Inhouse Storage System CDN Akamai Hosting Viewing Cache 5. Download HLS Files 1. RTMP restart Upload HLS Files Burst API Server 4. RESTART 3. Polling 2. RESTART event
- 39. Improved Hosting Performance GPUCPU 720p 1080p 220% 267%
- 40. Chat
- 41. Chat Overview CMS Hosting Viewing WebSocket CHAT API Servers JSON API JSON API JSON API WebSocket
- 42. Chat Overview Server 1 Server 2 Server 3 Room 1 Room 1 Redis Cluster Publish/Subscribe
- 43. Chat Overview Server 1 Server 2 Server 3 Room 1 Room 1 Room 2 Room 2 Redis Cluster Room 2
- 45. Issues
- 46. Issues Over 10,000 messages per minute
- 48. Love Clients Store (Increment) Flush Burst API Redis Cluster MySQL Clients - ranking (ZSET) - Total count (NUM) Data (log) Using Lua Script
- 49. “Love” Total Ranking Get API Redis Cluster Clients Using ZREVRANGE
- 50. “Love” Total Ranking Get (Archive) API Redis Cluster Clients MySQL Using ZREVRANGE
- 51. Example 1 11 million Loves in 120 minutes
- 52. Example 2 2.3 million Loves in 30 minutes
- 53. Chat and “Love” Issues Are Also Explained on our Blog http://developers.linecorp.com/blog/ja/?p=3653 http://developers.linecorp.com/blog/ja/?p=3771
- 54. Summary Provided overview of LINE LIVE Talked about video streaming methods Talked about implementation of traffic-inviting chat and “Love” features
- 55. Any Questions?