![[Spark Summit EU 2017] Apache spark streaming + kafka 0.10 an integration story](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkstreamingkafka0-171026072955-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to 8 Lessons Learned from Using Kafka in 1500 microservices - confluent streaming event (20)
Ad
More from Natan Silnitsky (20)
Ad
Recently uploaded (20)
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streaming event
- 1. natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 8 Lessons Learned from using Kafka with 1500 microservices Natan Silnitsky Backend Infra Developer, Wix.com
- 2. >180M registered users (website builders) from 190 countries 5% of all Internet websites run on Wix 4000+ people work at Wix >500B HTTP Requests / Day 6PB of static content At Wix @NSilnitsky
- 3. Kafka Messages per day Microservices (Service) Developers 300M 1510M 500 1,450 100 600 2017 2020 At Wix
- 4. What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 5. What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 6. #1 Common Infra with common features.What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 7. Kafka Consumer Kafka Producer Kafka Broker Greyhound Wraps Kafka Service A Service B @NSilnitsky
- 8. Kafka Consumer Kafka Producer Kafka Broker Greyhound Wraps Kafka Service A Service B @NSilnitsky
- 9. Kafka Consumer Kafka Producer Kafka Broker Service A Service B Abstract so that it is easy to change for everyone Simplify APIs, with additional features Greyhound Wraps Kafka
- 10. Kafka Consumer Kafka Producer Kafka Broker Service A Service B Scala ZIO + Java API Greyhound Wraps Kafka @NSilnitsky
- 11. Greyhound Wraps Kafka Simple Consumer API - Boilerplate @NSilnitsky
- 12. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 13. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 14. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 15. static void runConsumer() throws InterruptedException { String topic = "some-topic"; MessageHandler<SomeMessage> handler = message -> System.out.printf("Record value:%dn", message.messageValue); GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler); } Greyhound Consumer API * No explicit commit
- 17. static void runConsumer() throws InterruptedException { final Consumer<Long, SomeMessage> consumer = createConsumer(); while (true) { final ConsumerRecords<Long, SomeMessage> consumerRecords = consumer.poll(1000); consumerRecords.forEach(record -> { System.out.printf("Record value:%dn", record.value().messageValue); }); consumer.commitAsync(); } } Kafka Consumer API
- 18. static void runConsumer() { MessageHandler<PurgeSite> handler = ... GreyhoundConsumer consumer = GreyhoundConsumer.aGreyhoundConsumerSpec(topic, handler) .withGroup("some-group") .withMaxParallelism(40); } Greyhound Consumer @NSilnitsky
- 19. Kafka Broker Site Chat-bot-m essages Topic Greyhound Consumer Kafka Consumer SCALA ZIO FIBERS + QUEUES 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 (THREAD-SAFE) PARALLEL CONSUMPTION @NSilnitsky
- 20. + Retries! ...what about Error handling? Greyhound Wraps Kafka Simple Consumer API Thread-safe Parallel Consumption
- 21. static void setupConsumer() { ConsumeRetryPolicy retryPolicy = ConsumeRetryPolicy.aRetryPolicy( Retries.fromBackoffs(Duration.ofSeconds(1), Duration.ofMinutes(10), ...)); GreyhoundConsumer renewConsumer = GreyhoundConsumer.aGreyhoundConsumerSpec( topic, renewHandler) .withGroup("some-group") .withRetry(retryPolicy); } Greyhound Consumer @NSilnitsky
- 22. Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Greyhound Consumer Kafka Consumer FAILS TO READ @NSilnitsky
- 23. Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-0 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Inspired by Uber RETRY! Greyhound Consumer Kafka Consumer RETRY PRODUCER @NSilnitsky
- 24. #2 Retry Topics will cause your cluster to grow faster. 😐 What do you do when the traffic, meta-data, and amount of developers and use cases grow? #1 Common Infra
- 25. Kafka Broker renew-sub-topic-retry-0 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-N 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 RETRY! Inspired by Uber @NSilnitsky
- 26. Kafka Broker renew-sub-topic-retry-0 0 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 5 Kafka Broker renew-sub-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 renew-sub-topic-retry-N RETRY! Inspired by Uber 0 2 3 4 51 @NSilnitsky
- 27. Retries same message on failure * lag BLOCKING POLICY HANDLER Kafka Broker source-control- update-topic 0 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50 1 2 3 4 5 Greyhound Consumer Kafka Consumer build-log-service
- 29. + Context Propagation Super cool for us Greyhound Wraps Kafka Simple Consumer API Thread-safe Parallel Consumption Scheduled retries & blocking handler
- 30. CONTEXT PROPAGATION Language User Type USER REQUEST METADATA Sign up Site-Members Service Browser Geo ... @NSilnitsky
- 33. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #3 Self-service tooling and documentation. #2 Retry Topics - bigger cluster #1 Common Infra
- 34. Our team
- 36. 1. Kafka CLI scripts (+ flags) 2. Yahoo Kafka Manager (OSS) 3. Confluent Control Centre 4. A control plane of our own Self-service Tools
- 44. Self-service Docs How do I investigate this lag? How do I add retries on errors? 1. Github Readme for Greyhound code 2. Internal StackOverflow Q&A 3. Slack bot that answers without you
- 48. protobuf internal API site Self-service Docs Discover topics and message structure. Message structure defined in... And exposed with... @NSilnitsky
- 49. protobuf internal API site Self-service Docs Discover topics and message structure. Or… avro schema registry Message structure defined in... And exposed with... @NSilnitsky
- 50. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #4 Async event driven monitoring is less trivial. #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra
- 52. @NSilnitsky
- 53. #5 Proactive broker maintenance. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #4 Non-trivial Monitoring #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra
- 54. ● Add brokers when needed ● Split clusters when needed ● Delete unused topics ● Avoid hard failures As a rule of thumb, we recommend each broker to have up to 4,000 partitions and each cluster to have up to 200,000 partitions. https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions Don’t let brokers break... @NSilnitsky
- 55. We’re migrating to Confluent Cloud @NSilnitsky ● High Availability ● Don’t need to worry about scaling clusters
- 56. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #6 Avoid using Kafka SDK directly in nodeJs #5 Proactive broker maintenance #4 Non-trivial Monitoring #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra
- 57. Greyhound Kafka Broker Producer Consumer Scala services NodeJS services ConsumerProducer Single Event loop @NSilnitsky
- 58. Greyhound Kafka Broker Producer Consumer Scala services NodeJS services ConsumerProducer Single Event loop ✘ @NSilnitsky
- 59. * memory Greyhound Kafka Broker Consumer Sidecar (Or REST Proxy) gRPC NodeJS services Single Event loop
- 60. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #7 Consume and project #5 Proactive broker maintenance #4 Non-trivial Monitoring #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra #6 Avoid nodeJs SDK
- 62. Read + Writes Large MetaSite Object Request overload (~1M RPM requests) MetaSite Site installed Apps? RPC Wix Bookings Wix Restaurants Site version? Site owner? RPC RPC Wix Stores @NSilnitsky
- 63. Read + Writes Large MetaSite Object Request overload (~1M RPM requests) MetaSite Site installed Apps? RPC Wix Bookings Wix Restaurants Site version? Site owner? RPC RPC Wix Stores @NSilnitsky
- 65. Kafka Broker Filter Site installed Apps Updated! Installed Apps Context 1. Produce to Kafka 2. Consume and Project Producer MetaSite Site Updated! Consumer Reverse lookup writer
- 66. Kafka Broker Filter Site installed Apps Updated! Installed Apps Context 1. Produce to Kafka 2. Consume and Project Producer MetaSite Site Updated! Consumer Reverse lookup writer Reverse lookup reader RPC RPC RPC 3. Split Read from Write
- 67. Kafka messaging is event driven. It is only relevant to service-service communications, not for browser-server interactions, where a user is waiting, right? @NSilnitsky
- 68. Kafka messaging is event driven. It is only relevant to service-service communications, not for browser-server interactions, where a user is waiting, right? Wrong. @NSilnitsky
- 69. What do you do when the traffic, meta-data, and amount of developers and use cases grow? #8 WebSockets are Kafka’s best friend #7 Consume and project #5 Proactive broker maintenance #4 Non-trivial Monitoring #3 Self-service tooling & docs #2 Retry Topics - bigger cluster #1 Common Infra #6 Avoid nodeJs SDK
- 70. Completely distributed and event driven Kafka WebSockets + = @NSilnitsky
- 71. Kafka Broker Consumer Browser Producer Contacts Importer Contacts Jobs Web Sockets Service Use Case: Long-running async business process @NSilnitsky
- 72. Kafka Broker Subscribe for notifications ConsumerProducer Use Case: Long-running async business process Contacts Importer Contacts Jobs Browser Web Sockets Service @NSilnitsky
- 73. Kafka Broker Import CSVs ConsumerProducer Use Case: Long-running async business process Contacts Importer Contacts Jobs Browser Web Sockets Service @NSilnitsky
- 74. Web Sockets Service Kafka Broker Consumer * distributed Use Case: Long-running async business process Producer Contacts Jobs Browser done! Contacts Importer
- 75. So, What do you do when the traffic, meta-data, and amount of developers and use cases grow?
- 76. Wix Created an entire ecosystem to support large-scale Kafka-related needs.
- 77. A Java/Scala high-level SDK for Apache Kafka. 0.1 is out! github.com/wix/greyhound
- 78. Thank You natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil