Pipelining the Heroes with Kafka and Graph
2 likes436 views
Will Bleker and Gary Stewart of ING discuss pipelining data using Kafka and Neo4j Graph. They describe shifting from a traditional database approach of managing "trains" to a NoSQL approach of treating data as disposable "cattle". This allows rebuilding the data pipeline frequently for more agility. They provide an example architecture and lessons learned about simplifying operations and increasing resilience by treating data as immutable and designing systems to rebuild frequently.
![Simple model
• (:Source)-[:HAS_MSG]->(:Message)
Lots of nodes
31](https://image.slidesharecdn.com/kafkameetuppipelinetheheroesv3-181203233415/85/Pipelining-the-Heroes-with-Kafka-and-Graph-31-320.jpg)
![Consider traversing to reduce density
• (:Source)-[:LAST]->(:Message)->[:PREV]->(:Message)
Modify code
à Simply rebuild and J
Partitioning in graph
32](https://image.slidesharecdn.com/kafkameetuppipelinetheheroesv3-181203233415/85/Pipelining-the-Heroes-with-Kafka-and-Graph-32-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Pipelining the Heroes with Kafka and Graph (20)
Ad
More from confluent (20)
Ad
Recently uploaded (20)
Pipelining the Heroes with Kafka and Graph
- 1. Pipelining the Heroes with Kafka and Graph Will Bleker Gary Stewart Amsterdam, 12 November 2018
- 2. 2
- 3. Graph - Nodes and Relationships 3
- 4. Graph - Add properties 4
- 5. About us 5 EMPLOYED_BY EMPLOYED_BY Role: Chapter Lead Role: Platform Architect Gary Stewart Will Bleker
- 6. Market leaders Benelux Growth markets Commercial Banking Challengers 6 About ING Over 40 countries 52,000+ employees
- 7. Concepts 7 • Concepts • Pipeline • Use case • Learnings from NoSQL thinking
- 8. A little history … 8 Pets • Unique and indispensable • Often long lived • Scale-up • Active-passive architecture, pairs of servers, etc Cattle • Disposable, one of a herd • Often short lived • Scale-out • Active-active architecture • ‘Cattle’ datastores include • Apache Cassandra • Apache Kafka
- 9. Databases are much like the transport industry • Undergone massive transformations • Various types suitable for different needs • Trains • Relatively expensive • Built on-demand • Longer lifespan • Cars • Manual to mass production • Almost consumable • Often cheaper to rebuild, recycle than repair Our twist – trains versus cars 9
- 10. Our journey to NoSQL 10 A few years back we started with Cassandra • Exotic key/value store (column family) • Distributed active-active architecture • Tuneable consistency • Paradigm shift • Nights and weekends are now free (including LCM) • However now a liquid expectation! RDBMSNoSQL
- 11. Adding graph to the landscape 11 Gap in our DB landscape - connected data • Another NoSQL database! Interesting observation • Key/Value easy to mess up a partition • RDBMS easy to mess up a table • Graph easy to mess up the database! ? RDBMSNoSQL Graph
- 12. Typical architecture for caching data 12 SOR Challenges include • Initial load is usually once-off or manual • Lifecycle management (LCM) is hard and per component • Resilience patterns need to be applied everywhere • Cross DC resilience pattern also required Real-Time Sync Batch Loader Cache Cache
- 13. Cache ‘Cattle’ Paradigm 13 SOR Real-Time Sync Single Materialized ViewCommit Log Challenges include • Initial load Integrated by design • Lifecycle management (LCM) Simplified • Resilience Simplified • Cross DC Simplified Batch Loader
- 14. Cache ‘Cattle’ Paradigm – Taking it further 14 SOR Real-Time Sync Single Materialized ViewCommit Log E.g. Apache Kafka supports compacting topics • Ideal for rebuilding cache use cases Batch Loader
- 15. Cache ‘Cattle’ Paradigm – Going all the way 15 Real-Time Sync Single Materialized ViewCommit Log Batch Loader SOR
- 16. Pipeline 16 • Concepts • Pipeline • Use case • Learnings from NoSQL thinking
- 17. Pipelining the ‘Materialized View’ 17 Real-Time Sync Batch Loader Materialized View SOR
- 18. Block ports Get stuff • Artifacts Pre Processing • Convert data • neo4j-admin import Start Neo4j Post processing Create Indexes Complex calculations Start applications in desired order Unblock ports Switch - Opening for client requests Pipelining the ‘Materialized View’ 18 Real-Time Sync Batch Loader Materialized View SOR
- 20. Neo4j-admin import 20 IMPORT DONE in 4m 17s 923ms. Imported: 11 759 853 nodes 11 774 714 relationships 35 332 411 properties IMPORT DONE in 1m 41s 941ms.Imported: 26 444 678 nodes 26 444 020 relationships 66 129 151 properties
- 21. Use case 21 • Concepts • Pipeline • Use case • Learnings from NoSQL thinking
- 22. • Customer Teams • Platform Teams • Components Challenges • Rapid growth • Too many dependencies • Too many changes • Often converging • Too much data (metrics) • Heroes • … Change is needed! Managing our Cassandra Platform 22
- 23. Observation • Ops person ‘pulls’ information when needed (requires logging on) • Full shifting left is particularly challenging for multi-tenant platforms Pipeline everything • We don’t value initial bursts of energy anymore • Pipeline the heroes J • Design to rebuild ‘Personal’ goal • Learn Graph technology Understanding the challenges ahead 23
- 24. Architecture 24 • Simple and effective • Executes ‘Ops’ commands: • ps –ef • netstat • cat config.json • Raw output sent as message • All parsing rules for raw messages • Store in databases • Complex rules and analysis
- 25. Architecture Commit LogProducer Materialized view Consumer App 25
- 26. Consuming message (data pipeline) • Parse • Store as Key/Value • Calculate • Observe • Recommend Principles • Compare to inventory not ‘self discovery’ • Define agreements in code • Design to be idempotent as possible • No data migrations allowed (simply rebuild!) Consumer - Graph Model 26
- 27. Graph proven to be a VERY good fit for this use case! • Already producing 50+ commands • Heroes have a place to ‘query’ • Various unexpected findings • Misconfigured tenants • 100+ recommendations made Architecture now helps with • Allow us to be ‘in’ control without ‘having’ control • Remaining agile • Shifted left our operational analysis 300+ cars were destroyed in our 8 month journey to learn graph! So what happened? 27
- 28. Learnings from NoSQL thinking 28 • Concepts • Pipeline • Use case • Learnings from NoSQL thinking
- 29. Parse each message into key/value WITH {a: "val 1", b: "val 2", c: "val 3"} AS data CREATE (n:MyNode) SET n += data; Added 1 label, created 1 node, set 3 properties … Benefits • Easier to build insights with existing data • ‘Raw’ data remains unchanged • However if parsing rules change Modify code à Simply rebuild and … Breaking it down to Key/Value 29
- 30. 30
- 31. Simple model • (:Source)-[:HAS_MSG]->(:Message) Lots of nodes 31
- 32. Consider traversing to reduce density • (:Source)-[:LAST]->(:Message)->[:PREV]->(:Message) Modify code à Simply rebuild and J Partitioning in graph 32
- 33. 33
- 34. MATCH … OPTIONAL MATCH WHERE … MERGE MATCH … WHERE … OPTIONAL MATCH MERGE Modify code à Simply rebuild and J Attempted to migrate data anyway 34
- 35. 35
- 36. Kafka is not just a ‘messaging’ product Actually it is a distributed commit log! Like Graph we learnt Kafka too!! What about Kafka? 36
- 37. Conclusions 37
- 38. Summary 38 Shifting left • Requires initial extra effort • Design for rebuild instead of migrations • Creates a learning environment for new technology Ensures agility • Adding new features • Implement learnings quickly • Easier to experiment Ensures control • Easier to secure • Data quality actually improves due to frequent rebuilds • Safer and cheaper to build systems to not last long
- 39. Pipeline the heroes • Create a data pipeline • Automate data collection • Automate analysis Shift left your operational analysis! In other words … 39
- 40. Thank you [email protected] - @WBleker [email protected] - @Gaz_GandA We are hiring! 40
- 41. Follow us to stay a step ahead ING.com YouTube.com/ING SlideShare.net/ING@ING_News LinkedIn.com/company/ING Flickr.com/INGGroupFacebook.com/ING
- 42. ING Group’s Annual Accounts are prepared in accordance with International Financial Reporting Standards as adopted by the European Union (‘IFRS-EU’). In preparing the financial information in this document, the same accounting principles are applied as in the 2014 ING Group Annual Accounts. All figures in this document are unaudited. Small differences are possible in the tables due to rounding. Certain of the statements contained herein are not historical facts, including, without limitation, certain statements made of future expectations and other forward-looking statements that are based on management’s current views and assumptions and involve known and unknown risks and uncertainties that could cause actual results, performance or events to differ materially from those expressed or implied in such statements. Actual results, performance or events may differ materially from those in such statements due to, without limitation: (1) changes in general economic conditions, in particular economic conditions in ING’s core markets, (2) changes in performance of financial markets, including developing markets, (3) consequences of a potential (partial) break- up of the euro, (4) the implementation of ING’s restructuring plan to separate banking and insurance operations, (5) changes in the availability of, and costs associated with, sources of liquidity such as interbank funding, as well as conditions in the credit markets generally, including changes in borrower and counterparty creditworthiness, (6) the frequency and severity of insured loss events, (7) changes affecting mortality and morbidity levels and trends,(8) changes affecting persistency levels, (9) changes affecting interest rate levels, (10) changes affecting currency exchange rates, (11) changes in investor, customer and policyholder behaviour, (12) changes in general competitive factors, (13) changes in laws and regulations, (14) changes in the policies of governments and/or regulatory authorities, (15) conclusions with regard to purchase accounting assumptions and methodologies, (16) changes in ownership that could affect the future availability to us of net operating loss, net capital and built-in loss carry forwards, (17) changes in credit ratings, (18) ING’s ability to achieve projected operational synergies and (19) the other risks and uncertainties detailed in the Risk Factors section contained in the most recent annual report of ING Groep N.V. Any forward-looking statements made by or on behalf of ING speak only as of the date they are made, and, ING assumes no obligation to publicly update or revise any forward-looking statements, whether as a result of new information or for any other reason. This document does not constitute an offer to sell, or a solicitation of an offer to purchase, any securities in the United States or any other jurisdiction. The securities of NN Group have not been and will not be registered under the U.S. Securities Act of 1933, as amended (the “Securities Act”), and may not be offered or sold within the United States absent registration or an applicable exemption from the registration requirements of the Securities Act. www.ing.com Disclaimer 42