Modern Data Flow
1 like276 views
The document discusses principles of modern data flow, including: - Streaming to enable real-time data processing - Decentralized data pipelines and data streaming as the central nervous system - Declarative APIs and SQL for data in motion to simplify development - Developer-oriented approaches with code and open platforms at the forefront - Governance including catalogs for discovery, schemas as APIs, and lineage for observability The principles are positioned as an evolution from older batch, centralized, infrastructure-heavy approaches to real-time, distributed, developer-centric data streaming architectures. Governance is discussed as important for enabling innovation while maintaining security and control.
![47
Schemas
Become the “API”
for Data Flow
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Product",
"description": "A product from Acme's catalog",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a
product",
"type": "integer"
},
"name": {
"description": "Name of the product",
"type": "string"
},
"price": {
"type": "number",
"minimum": 0,
"exclusiveMinimum": true
}
},
“required": ["id", "name", "price"]
}](https://image.slidesharecdn.com/02202210dimkeynotemoderndataflow-221020122224-46d9b1a0/85/Modern-Data-Flow-47-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Modern Data Flow (20)
Ad
More from confluent (20)
Ad
Recently uploaded (20)
Modern Data Flow
- 1. Modern Data Flow Data Pipelines Done Right Kai Waehner Field CTO kai.waehner@confluent.io linkedin.com/in/kaiwaehner @KaiWaehner kai-waehner.de
- 2. 2 Huge Growth in the Community >100,000+ Organizations Using Kafka >41,000 Kafka Meetup Attendees >32,000 Stack Overflow Questions >12,000 Jiras for Apache Kafka
- 3. 3 Huge Growth in Usage Active Monthly Unique Users Kafka Java Client Library Source: Sonatype Kafka JAN 2017 JAN 2018 JAN 2019 JAN 2020 JAN 2021 JAN 2022 0 200,000 400,000 600,000 800,000
- 4. 4 Growing Demand 31,000 Open Job Listings Request Kafka Skills 4
- 5. Real-time Data beats Slow Data. Logistics Real-time sensor diagnostics Delivery planning Estimated time of arrival updates Payment Fraud detection Risk systems Mobile applications / customer experience Retail Real-time inventory Real-time point of sale reporting Personalization Sales Real-time recommendations Personalized coupon feed Pay by walking out
- 6. Major New Platform Around Data in Motion 6
- 7. Data Streaming as Central Nervous System MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Supplier Alert Forecast Inventory Customer Order 7
- 8. 8 Credit Card Fraud Detection Deidentification Personal Recommendations Shipment Tracking / Alerting JIT Inventory Logistics Geofencing Route Optimization Payment Verification Dynamic Pricing APT Detection Customer 360 Cloud DWH Ingestion Clickstream Log Aggregation Real-Time Analytics Ingestion Real-Time Analytics Change Data Capture Complex Events Processing Hybrid / Multicloud Data Flow Saas App Integration Streaming ETL Ml Feature Pipelines Customer Retention / Loyalty Fleet Management
- 9. Credit Card Fraud Detection Customer Retention / Loyalty Deidentification Payment Verification Fleet Management Shipment Tracking / Alerting JIT Inventory Logistics Geofencing Route Optimization Personal Recommendations Dynamic Pricing APT Detection Real-Time Analytics Hybrid / Multicloud Data Flow Customer 360 Clickstream Ml Feature Pipelines Cloud DWH Ingestion Real-Time Analytics Ingestion Change Data Capture Log Aggregation Complex Events Processing Streaming ETL Saas App Integration Applications Pipelines Streaming Platform 9
- 10. 10 Data Pipeline Technologies Old and New AWS Glue
- 11. 11 Today’s data pipelines: Kafka is necessary but not sufficient.
- 12. 12 Principles of Modern Data Flow Governed & Observable Streaming Decentralized Declarative Developer-Oriented
- 13. 13 Principles of Modern Data Flow: Streaming
- 15. 15 Would you blindly cross the street with traffic information that is 5 minutes old? 15
- 16. Real-time Data beats Slow Data. Manufacturing Sensor diagnostics MES/ERP Integration Reporting Edge Computing Condition Monitoring Predictive Maintenance Quality Assurance Logistics Supply Chain Inventory management Track & Trace Context-specific routing Cybersecurity Threat detection Intrusion detection Incident response Military decisions
- 18. 18 You Can (Increasingly) Have Your Cake and Eat It Too
- 20. 20 Streaming is a Generalization of Batch Streaming Batch
- 21. Pipeline for Legacy and Modern APIs @ Allianz 21
- 22. 22 Principles of Modern Data Flow: Decentralized
- 23. Centralized Data Warehouse ETL Classic ETL Classic Data Warehousing 23
- 24. 24
- 26. Publish/Subscribe, in the Large 26 Customer Topic Analysis Customer Support System CRM Personalization & Marketing Systems
- 28. Publish/Subscribe, in the Large 28 Customer Topic Analysis Customer Support System CRM Personalization & Marketing Systems
- 30. Data Mesh @ Raiffeisen Bank International 30
- 31. 31 Principles of Modern Data Flow: Declarative
- 32. 32 Say what you mean.
- 33. 33 “Everything should be made as simple as possible, but no simpler.” Albert Einstein
- 34. 34 SQL provides this for Data at Rest.
- 35. 35 SQL provides this for Data in Motion.
- 36. 36 Pages Per Minute Sessions Errors Per Minute Connector -- number of errors per min, using 'HAVING' Filter to show ERROR codes > 400 -- where count > 5 CREATE TABLE errors_per_min_alert WITH (KAFKA_TOPIC = 'errors_per_min_alert') AS SELECT status AS k1, AS_VALUE(status) AS status, WINDOWSTART AS EVENT_TS, COUNT(*) AS errors FROM clickstream WINDOW HOPPING (SIZE 60 SECOND, ADVANCE BY 20 SECOND) WHERE status > 400 GROUP BY status HAVING COUNT(*) > 5 AND COUNT(*) IS NOT NULL; -- Enriched user details table: -- Aggregate (count&groupBy) using a TABLE-Window CREATE TABLE user_ip_activity WITH (KEY_FORMAT = 'JSON', KAFKA_TOPIC = 'user_ip_activity') AS SELECT username AS k1, ip AS k2, city AS k3, AS_VALUE(username) AS username, WINDOWSTART AS EVENT_TS, AS_VALUE(ip) AS ip, AS_VALUE(city) AS city, COUNT(*) AS count FROM user_clickstream WINDOW TUMBLING (SIZE 60 SECOND) GROUP BY username, ip, city HAVING COUNT(*) > 1;
- 37. Fraud Detection @ Kakao Games Detecting and operation anomalies with 300+ patterns through KSQL, including bonus abuse, multiple account usage, account takeover, chargeback fraud, affiliate fraud...
- 38. 38 Principles of Modern Data Flow: Developer-Oriented
- 39. Data Pipeline Technologies Old and New AWS Glue 39
- 40. Code is King Development is About Evolution Open Platforms Win 40
- 41. Variety of streaming APIs @ Big Commerce 41 Bot Filter
- 43. 43 Principles of Modern Data Flow: Governed & Observable
- 44. 44 Two Conflicting Pressures On Organizations Unlock the data to enable innovation Lock up the data to keep it safe
- 45. 45 Three Key Tools For Governing Data Flows Where to get the data streams Catalog How the data streams got here Lineage What the data streams looks like Schema
- 46. The catalog makes your streams discoverable. 46
- 47. 47 Schemas Become the “API” for Data Flow { "$schema": "http://json-schema.org/draft-04/schema#", "title": "Product", "description": "A product from Acme's catalog", "type": "object", "properties": { "id": { "description": "The unique identifier for a product", "type": "integer" }, "name": { "description": "Name of the product", "type": "string" }, "price": { "type": "number", "minimum": 0, "exclusiveMinimum": true } }, “required": ["id", "name", "price"] }
- 48. 48 Lineage lets you observe your real-time data flow graph.
- 49. 49 These governance features must be designed for a decentralized, streaming, developer-oriented world.
- 50. 50 Governance @ Raiffeisen Bank International (RBI)
- 51. 51 Principles of Modern Data Flow Governed & Observable Streaming Decentralized Declarative Developer-Oriented
- 52. 52 Ungoverned & Dark Batch Centralized Infrastructure-Heavy GUI-Oriented Governed & Observable Streaming Decentralized Declarative Developer-Oriented
- 53. Thank You