Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandran, Priceline.com) Kafka Summit NYC 2019
3 likes864 views
This document discusses using Kafka Streams to transform operational metrics data from Priceline applications before loading it into Splunk. It describes how the legacy monitoring system worked, and the motives for moving to Kafka and Kafka Streams. It then explains how data is collected from applications into Kafka topics, and how various transformations like formatting, key application, and aggregation are performed using Kafka Streams before loading to Splunk. It also discusses testing, monitoring, and debugging Kafka Streams applications.
Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandran, Priceline.com) Kafka Summit NYC 2019
- 1. Data Transformations on Operational Metrics using Kafka Streams Vidhya Ramachandran Mukund Murthy
- 2. About Priceline Priceline.com is part of the Booking Holdings Group. The Booking Holdings Group is the world leader in online travel & related services. The Group’s mission is to help people experience the world.
- 3. About Us Vidhya Ramachandran Mukund Murthy Director Engineering, Common Platform Services Software Engineer , Common Platform Services
- 5. Legacy Architecture Custom Alerting System Custom Dashboards Application Servers
- 6. Kafka - The Only Constant!
- 8. Motive behind going with Kafka & Kafka Streams
- 9. Kafka Infrastructure & new Monitoring Solution & other Sinks Config Reader Event Listener Applications Message Router Embedded Agent Asynchronous Message Dispatcher Kafka Consumer Cluster Infrastructure Code embedded in the Products to Produce Events to the Kafka Topic Splunk HEC Add other sinks Data Collection Console
- 10. Priceline Data Collection Console
- 11. Priceline Streams Multiplexing • Stream = Events emitted by some source (for e.g. a Priceline Application) • Multiple Streams can come from the same Application and be written into the same topic • A stream can be emitted by multiple Applications Stream 1 Stream 3 Stream 2 Topic
- 12. Priceline Data Collection Console
- 13. Streams in Splunk before & after transformations
- 14. License Reduction from Transformations Operational metrics data flowing through Kafka are converted to Pipe separated format and the keys are applied back as search time extractions.
- 17. Topology – Kafka Streams
- 18. Windowed Events
- 21. License Reduction from Summarizations
- 22. Late Arriving Events Windowed Event Entries | T1 | T2 | T3 | …… |1 2 3 |5 6 7 |9 10 11 | --------> 4 (T1) --------- > 8 (T2) Key Value T1 1,2,3 T1 1,2,3,4 T2 5,6,7 T2 5,6,7,8 T3 9,10,11
- 23. Processing too long… Measure & make sure to keep your Processors simple and straightforward Maintain SLA in Monitoring system < 5seconds from Event Time…
- 25. Gotchas…
- 26. Testing Kafka Streams • Unit testing individual processors and transformers • Integration testing with an embedded Kafka cluster, multiple instances • Integration Testing the aggregation by repeating the concatenated raw messages in the final aggregated event
- 28. 28 • Splunk Add-on to Poll local or remote JMX management Servers • Index MBean attributes, outputs from MBean operations, and MBean notifications. • Configurable Templates to selectively monitor JMX stats Kafka Monitoring – JMX Stats
- 29. Kafka Monitoring - Dashboards
- 30. Kafka Monitoring - Alerts
- 31. Monitoring Kafka Streams • Thread metrics • Average time for commits, poll, process operations • Tasks created per second, tasked closed per second • Task metrics • Average number of commits per second • Average commit time • Processor node metrics • Average and max processing time • Average number of process operations per second Forward rate • State store metrics • Average execution time for put, get, and flush operations • Average number put, get, and flush operations per second
- 33. Conclusion & Next Steps
- 34. Acknowledgements • Our Core Platform team for their contributions to the Streaming library and Data Collection Console • To Confluent support and sales team
- 35. We are Hiring!!! https://careers.priceline.com/
- 36. Questions