Netflix Keystone—Cloud scale event processing pipeline
8 likes3,362 views
The document discusses the evolution and architecture of Netflix's Keystone pipeline, designed for cloud-scale event processing, highlighting its daily processing capabilities reaching over 1 trillion events. It emphasizes the culture at Netflix promoting autonomy in operations and open-source contributions, showcasing the pipeline's innovations and performance metrics. Additionally, it touches upon the future of stream processing at Netflix with plans for integration of new technologies like Apache Flink and Beam.
![A NETFLIX ORIGINAL SERVICE
Netflix Keystone - Cloud scale event processing pipeline
[ O’reilly - Turning big data into knowledge ]
Monal Daxini](https://image.slidesharecdn.com/orielly-data-pipeline-webcast-160913131854/85/Netflix-Keystone-Cloud-scale-event-processing-pipeline-1-320.jpg)
![SPaaS Architecture
SPaaS
Manager
Container
Runtime
Beam API
or
Framework Specific API
[ Dockerized Job ]
1. Create 2. Submit 3. Launch
Runner
Flink / Spark /
Mantis
Running
Job
1. Submit Job
DSL (SQL)
Job Dashboard
BETA](https://image.slidesharecdn.com/orielly-data-pipeline-webcast-160913131854/85/Netflix-Keystone-Cloud-scale-event-processing-pipeline-77-320.jpg)
Netflix Keystone—Cloud scale event processing pipeline
- 1. A NETFLIX ORIGINAL SERVICE Netflix Keystone - Cloud scale event processing pipeline [ O’reilly - Turning big data into knowledge ] Monal Daxini
- 2. ● Season 1 ○ Keystone pipeline - Why? How? What? ● Season 2 ○ Trailer What to expect?
- 3. Monal Daxini I lead the Stream Processing effort in the Real Time Data Infrastructure team @ Netflix @monaldax #Netflix #Keystone Who?
- 4. Netflix Is a Data Driven Company Content Product Marketing Finance Business Development Talent Infrastructure ←CultureofAnalytics→
- 5. Netflix Service Truly Global Internet TV Network
- 6. Over 80 Million Members 190 Countries 1000+ devices
- 7. 125,000,000 hours/day 14,269 years worth / day 37% of Internet traffic at peak
- 8. Over 1,000,000,000,000 That’s a huge number! Keystone Events Processed Every Day
- 9. Events Trend 1/2014 80B / day 1/2015 300B / day 1/2016 1T / day
- 10. Keystone Scale Daily Averages ● 700B unique events ingested ● 1T events processed ● 1.5PB / day ● 4K / event Peak ● 1T unique events ingested/day ● 12.5M / sec ● 35GB / sec ● 10MB / message
- 11. ● 99.99% / Four 9s ● Initial numbers with VPC ○ 99.9999% / Six 9s Keystone Availability
- 13. In the beginning… Chukwa EMR Event Producers
- 14. Q4 2014 - Chukwa / Suro Event Producer Druid Stream Consumers EMR Consumer Kafka Suro Router Event Producer Suro Kafka Suro Proxy
- 15. ● Support at-least-once processing ● Scale, Multi-tenancy, Ease of Operations ● Enable future value adds - Stream Processing As a Service ● Replace dormant open source software - Chukwa Why a new pipeline?
- 16. Goal - Migrate 1.3 PB of event data to a new Pipeline in flight, while not losing more that 0.1% of them
- 17. Q4 2015 - Keystone Stream Consumers Samza Router EMR Fronting Kafka Consumer Kafka Control Plane Event Producer KSProxy
- 18. Want to know more... Netflix Tech Blog - Pipeline Evolution
- 19. Event flow Keystone Pipeline As a Service
- 24. Self Service UI (Keystone Management)
- 27. Event Payload is Immutable At-least-once semantics* * Once the event makes it to Kafka, there are disaster scenarios where this breaks.
- 28. Injected Event Metadata ● GUID ● Timestamp ● Host ● App `
- 29. Custom Extensible Wire Protocol ● Backwards and forwards compatibility ● Support multiple serialization formats ○ JSON, AVRO, Protobuf in the works ● Additional metadata ● Efficient - 10 bytes overhead per message
- 30. Netflix Kafka Producer ● Configurable - topic to Kafka clusters routing ● Sticky partitioner ● Prefer event drop than disrupt producer app ● Best effort delivery, ack = 1 ● Buffer size tuning based on traffic
- 31. Kafka Clusters
- 33. Kafka (prod) Footprint Fronting Kafka Front Standby Kafka Consumer Kafka Number of Clusters 24 24 8 Number of Instances 3000+ 72 1000+ Retention Period 8 to 24 hrs 1 hr 2 to 4 hrs
- 34. ● Independent zookeeper cluster per Kafka cluster ● 5 nodes per ensemble - 160 zookeeper nodes ● 3 ASGs per cluster, 1 ASG per zone Kafka (prod) Footprint
- 35. ● Pioneer Tax ● Started with 0.7, went live with 0.8.2 ● Done moving to 0.9 & VPC ● Work closely with Confluent to get patches through ○ OSS contributions Kafka in the Cloud
- 36. ● No dynamic topic creation ● Two copies ● Rack / Zone aware partition assignment ● Per Cluster Stay under 10k partitions & 200 brokers ● Leave approx. 40% free disk space on each broker Fronting Kafka Topics
- 37. Want to know more... Netflix Tech Blog - Kafka in Keystone Pipeline
- 38. Routing Service
- 40. Samza Job Deployment ● Multiple Samza jobs for one Kafka source topic ● Each job processes messages for one sink ● Each job processes partitions only from one topic ● One checkpoint topic per Kafka source topic and multiple samza jobs ● Job starts with Immutable Config
- 41. ● Over 13,000+ docker containers (samza jobs) ● 1,300+ AWS C3-4XL instances Routing Service Footprint S3 ElasticSearch Consumer Kafka Number of containers 7000+ 1500+ 4500+
- 42. Routing Latency Fronting Kafka to Sinks S3 ElasticSearch Consumer Kafka 1 sec 13 sec 800 ms
- 44. Router Job Manager (Control Plane) EC2 Instances Zookeeper (Instance Id assignment) Job Job Job ksnode Checkpointing Cluster ASG
- 45. Custom Go Executor ./runJob Logs Snapshots Attach Volumes ./runJob ./runJob Reconcile Loop - 1 min Health Check What’s running in ksnode? Zookeeper (Instance Id assignment)
- 46. Logs ZFS Volume Snapshots Custom Go Executor . /runJo b . /runJo b . /runJo b Go Tools Server Client Tools Stream Logs Browse through rotated logs by date Ksnode Tooling
- 47. Yes! You inferred right! No Mesos & No Yarn
- 48. Samza Tweaks to ver 0.9.1 ● Using ThreadJobFactory - Simplifies deployment and reduces overhead ● SAMZA-41 - range based static partition range assignment ● SAMZA-775- size based Prefetch buffer ○ Default was count based (OOM), and not bytes based ○ Set per topic per job to 60 * peak bytes / sec over the past week.
- 49. Samza Tweaks to ver 0.9.1 ● Backported from 0.10 ○ SAMZA-655 - environment variable configuration rewriter ■ Pass config from RDS to executor to Docker to Samza Job ○ SAMZA-540 - expose latency related metrics in OffsetManager ■ checkpointed offset gauge
- 50. More Info - Monal’s Samza Meetup Slides (10/2015) Netflix Samza ver 0.9.1 Contributions
- 53. Customer Facing per topic end-to-end dashboard
- 54. Birth of Kafka Kong
- 55. A True Story Keystone went live 10.27.2015 2 days later...
- 56. A True Story ● 80% Loss over 6 hour period ● Large Kafka clusters were impacts, smaller ones were fine
- 57. ● At times things go wrong, and there’s no turning back ● Reduce complexity ● Minimize blast radius ● Find a way to start over fresh Lessons Learned
- 59. ● Cold standby Kafka cluster with 3 instances and different instance type ● Different ZooKeeper cluster with no state ● When failover occur ○ Scaling up cluster ○ Creating topic ○ Creating new routing jobs for failover cluster ○ Switch producer traffic! Failover
- 60. Kafka Kong At least once a week
- 61. Fronting Kafka Failover Self Service Tool
- 62. Fronting Kafka Failover Fully Automated ● Time is the essence - failover as fast as 5 minutes
- 63. Culture What does it have to do with building a pipeline?
- 64. "It may well be the most important document ever to come out of the Valley." 1 Sheryl Sandberg COO, Facebook 1 Business Insider, 2013
- 65. Netflix Culture Deck Netflix Culture Freedom & Responsibility
- 66. Open source and community participation is an integral part of our strategy and culture
- 67. Not DevOps, but move towards NoOps You build it! You run it!
- 68. My team has ● No dedicated product or project managers ● No separate devops or operations team We build and run what you saw today!
- 69. ● This does not mean we are constantly overworked ○ we make wise and simple choices and ○ lean towards automation & self-healing systems We build and run what you saw today!
- 70. We built a pipeline in a year with A very small team, Relevant new technology, Contributed back to OSS, and Processed over 1 Trillion messages / day Culture Impact
- 71. Season 2 Trailer
- 72. Create DuploⓇ Blocks: Let reusability drive new value Our Philosophy
- 73. Evolution Keystone Stream Processing (SPaaS) Keystone Management Keystone Messaging Schema Support
- 74. Simple and intuitive interface to manage all Keystone services Keystone Management
- 75. Keystone Stream Processing Stream Processing As a Service (SPaaS) Multi-tenant polyglot support for stream processing engines BETA
- 76. Big Data Systems - streaming Data Pipeline & Stream processing - Keystone - Samza / Flink (poc) Playback & edge Operations insight - Mantis Stream Processing - Spark Streaming * Metrics & monitoring - Atlas *
- 77. SPaaS Architecture SPaaS Manager Container Runtime Beam API or Framework Specific API [ Dockerized Job ] 1. Create 2. Submit 3. Launch Runner Flink / Spark / Mantis Running Job 1. Submit Job DSL (SQL) Job Dashboard BETA
- 78. ● Starting out Proof Of Concept with Apache Flink ● Exploring Apache Beam SPaaS - init( ) BETA
- 79. Why Apache Beam? ○ Portable API layer for building sophisticated data processing applications ○ Unified APi for processing bounded and unbounded data sources ○ Google lineage - Dataflow model, and reflects Google’s current work SPaaS - “Beam Me Up, Scotty ! "
- 80. Why Flink? ○ Flink implements the dataflow model ○ Correctness of results and powerful features for reasoning about time ○ Checkpoints, exactly-once processing ○ Event time, processing time, watermarks, triggers, aligned windows (fixed, sliding), unaligned windows (dynamic or session windows) ○ Flink’s core is a streaming engine SPaaS - Flink
- 81. More brain food... Netflix OSS Samza Meetup Presentation Netflix Tech Blog Spark Summit 2015 Talk
- 82. Thanks! Q & A You can find me at @monaldax [email protected]
- 83. ● Special thanks to all the people who made and released these awesome resources for free: ○ Photographs by Unsplash Credits