Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Uber Real Time Data Analytics (20)
Ad
Recently uploaded (20)
Uber Real Time Data Analytics
- 1. Real Time Data Analytics @ UberAnkur Bansal Apache Big Data Europe November 14, 2016
- 2. About Me ● Sr. Software Engineer, Streaming Team @ Uber ○ Streaming team supports platform for real time data analytics: Kafka, Samza, Flink, Pinot.. and plenty more ○ Focused on scaling Kafka at Uber’s pace ● Staff software Engineer @ Ebay ○ Build & scale Ebay’s cloud using openstack ● Apache Kylin: Committer, Emeritus PMC
- 3. Agenda ● Real time Use Cases ● Kafka Infrastructure Deep Dive ● Our own Development: ○ Rest Proxy & Clients ○ Local Agent ○ uReplicator (Mirrormaker) ○ Chaperone (Auditing) ● Operations/Tooling
- 5. Stream Processing Real-time Price Surging SURGE MULTIPLIERS Rider eyeballs Open car information KAFKA
- 6. Real-time Machine Learning - UberEats ETD
- 8. ● Fraud detection ● Share my ETA And many more ...
- 9. Apache Kafka is Uber’s Lifeline
- 10. Kafka ecosystem @ Uber
- 11. 100s of billion 100s TB Messages/day bytes/day Kafka cluster stats Multiple data centers
- 12. Kafka Infrastructure Deep Dive
- 13. Requirements ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
- 15. Kafka Pipeline: Data Flow 1 2 3 5 7 64 8
- 17. Kafka Clusters ● Use case based clusters ○ Data (async, reliable) ○ Logging (High throughput) ○ Time Sensitive (Low Latency e.g. Surge, Push notifications) ○ High Value Data (At-least once, Sync e.g. Payments) ● Secondary cluster as fallback ● Aggregate clusters for all data topics.
- 18. Kafka Clusters ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
- 20. Why Kafka Rest Proxy ? ● Simplified Client API ● Multi-lang support (Java, NodeJs, Python, Golang) ● Decouple client from Kafka broker ○ Thin clients = operational ease ○ Less connections to Kafka brokers ○ Future kafka upgrade ● Enhanced Reliability ○ Primary & Secondary Kafka Clusters
- 21. Kafka Rest Proxy: Internals
- 22. Kafka Rest Proxy: Internals
- 23. Kafka Rest Proxy: Internals ● Based on Confluent’s open sourced Rest Proxy ● Performance enhancements ○ Simple http servlets on jetty instead of Jersey ○ Optimized for binary payloads. ○ Performance increase from 7K* to 45-50K QPS/box ● Caching of topic metadata. ● Reliability improvements* ○ Support for Fallback cluster ○ Support for multiple Producers (SLA based segregation) ● Plan to contribute back to community *Based on benchmarking & analysis done in Jun ’2015
- 24. Rest Proxy: performance (1 box) Message rate (K/second) at single node End-endLatency(ms)
- 25. Kafka Clusters + Rest Proxy ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
- 27. Client Libraries ● Support for multiple clusters. ● High Throughput ○ Non-blocking, async, batching ○ <1ms produce latency for clients ○ Handles Throttling/BackOff signals from Rest Proxy ● Topic Discovery ○ Discovers the kafka cluster a topic belongs ○ Able to multiplex to different kafka clusters ● Integration with Local Agent for critical data
- 28. Client Libraries Add Figure What if there is network glitch / outage?
- 30. Kafka Clusters + Rest Proxy + Clients ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
- 32. Local Agent ● Local spooling in case of downstream outage/backpressure ● Backfills at the controlled rate to avoid hammering infrastructure recovering from outage ● Implementation: ○ Reuses code from rest-proxy and kafka’s log module. ○ Appends all topics to same file for high throughput.
- 34. Local Agent in Action Add Figure
- 35. Kafka Clusters + Rest Proxy + Clients + Local Agent ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
- 38. CONFIDENTIAL >> INSERT SCREENSHOT HERE << Mirrormaker : existing problems ● New Topic added ● New partitions added ● Mirrormaker bounced ● New mirrormaker added
- 39. uReplicator: In-house solution Zookeeper Helix MM Controller Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition MM worker1 MM worker2 MM worker3
- 40. uReplicator Zookeeper Helix MM Controller Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition Helix Agent Thread 1 Thread N Topic-partition MM worker1 MM worker2 MM worker3
- 41. Kafka Clusters + Rest Proxy + Clients + Local Agent ● Scale to 100s Billions/day → 1 Trillion/day ● High Throughput ( Scale: 100s TB → PB) ● Low Latency for most use cases(<5ms ) ● Reliability - 99.99% ( #Msgs Available /#Msgs Produced) ● Multi-Language Support ● Tens of thousands of simultaneous clients. ● Reliable data replication across DC
- 42. uReplicator ● Running in production for 1+ year ● Open sourced: https://github.com/uber/uReplicator ● Blog: https://eng.uber.com/ureplicator/
- 43. Chaperone - E2E Auditing
- 45. CONFIDENTIAL >> INSERT SCREENSHOT HERE << Chaperone : Track counts
- 46. CONFIDENTIAL >> INSERT SCREENSHOT HERE << Chaperone : Track Latency
- 47. Chaperone ● Running in production for 1+ year ● Planning to open source in ~2 Weeks
- 49. Why do we need it? 1 2 3 5 7 64 8 ● Most of infrastructure tuned for high throughput ○ Batching at each stage ○ Ack before produce (ack’ed != committed) ● Single node failure in any stage leads to data loss ● Need a reliable pipeline for High Value Data e.g. Payments
- 50. How did we achieve it? ● Brokers: ○ min.insync.replicas=2, can only torrent one node failure ○ unclean.leader.election= false, need to wait until the old leader comes back ● Rest Proxy: ○ Partition Failover ● Improved Operations: ○ Replication throttling, to reduce impact of node bootstrap ○ Prevent catching up nodes to become ISR
- 53. Partition Rebalancing ● Calculates partition imbalance and inter-broker dependency. ● Generates & Executes Rebalance Plan. ● Rebalance plans are incremental, can be stopped and resumed. ● Currently on-demand, Automated in the future.
- 55. Summary: Scale ● Kafka Brokers: ○ Multiple Clusters per DC ○ Use case based tuning ● Rest Proxy to reduce connections and better batching ● Rest Proxy & Clients ○ Batch everywhere, Async produce ○ Replace Jersey with Jetty ● XFS
- 56. Summary: Reliability ● Local Agent ● Secondary Clusters ● Multi Producer support in Rest Proxy ● uReplicator ● Auditing via Chaperone
- 57. Future Work ● Open source contribution ○ Chaperone ○ Toolkit ● Data Lineage ● Active Active Kafka ● Chargeback ● Exactly once mirroring via uReplicator
- 59. Extra Slides
- 60. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 Broker 3 100 101 Leader Committed Producer Acked
- 61. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 Broker 3 100 101 Leader Committed Producer Failed Acked
- 62. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 Broker 3 100 101 Leader Committed Producer
- 63. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 104 105 106 Broker 3 100 101 104 105 Leader Committed Producer Old HW
- 64. Kafka Durability (acks=1) Broker 1 100 101 102 103 Broker 2 100 101 104 105 106 Broker 3 100 101 104 105 Leader Committed Producer X Old HW X
- 65. Kafka Durability (acks=1) Broker 1 100 101 104 105 106 Broker 2 100 101 104 105 106 Broker 3 100 101 105 106 Leader Committed Producer data loss!!
- 66. Distributed Messaging system * Supported in Kafka 0.8+ ● High throughput ● Low latency ● Scalable ● Centralized ● Real-time
- 67. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Broker 2 Broker 3 ZooKeeper
- 68. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Partition 0 Broker 2 Partition 1 Broker 3 Partition 2 ZooKeeper
- 69. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Partition 0 Partition 2 Broker 2 Partition 1 Partition 0 Broker 3 Partition 2 Partition 1 ZooKeeper
- 70. What is Kafka? ● Distributed ● Partitioned ● Replicated ● Commit Log Broker 1 Partition 0 0 1 2 3 Partition 2 0 1 2 3 Broker 2 Partition 1 0 1 2 3 Partition 0 0 1 2 3 Broker 3 Partition 2 0 1 2 3 Partition 1 0 1 2 3 ZooKeeper
- 71. Kafka Concepts