Data lineage
- 1. Building and Scaling Data Lineage at Netflix Strata SF 2019 Di Lin, Girish Lingappa, Jitender Aswani (@jaswani)
- 2. CTA: Intentionally Purposeful to Improve Diversity in Tech
- 3. People, Security, & Infra Data Eng (DEI) Data Lineage Asset Inventory Anomaly Detection Leverage data to improve cloud infra
- 5. in thinking / planning stages?
- 9. #1 Data Scientist validate source of the metrics?
- 10. #2 Software Engineer who downstream to you service is impacted?
- 11. #3 Reliability Engineer identify and alert on jobs / table at risk of missing SLA?
- 12. Detection and Data Cleansing Use Cases Inspired Lineage Retention and Data Efficiency Cost Attribution Platform Reliability Data Integrity
- 14. Freedom & Responsibility distributed decision making => ● move fast, ● scale complexity, ● Judicious about what’s best for Netflix
- 15. Data Landscape
- 16. Top Challenges
- 17. Challenge # 1 Diverse Data Landscape First time expanding beyond traditional data warehouse realm
- 18. Challenge # 2 Platform Evolution New Platforms Upgrades
- 19. Challenge # 3 Data Conformance Variance in granularity of the meta data
- 21. Design Principle # 1 Ensure data coverage
- 22. Design Principle # 2 Enable seamless integration
- 23. Design Principle # 3 Envision a flexible data model
- 24. Data Coverage
- 25. Data Coverage - data gets pushed ● Platform tools publish events (preferred approach) ○ Compute engines ○ Data movement tools ○ Reporting tools
- 26. Data Coverage - we pull & parse data ● Ingestion scripts parse logs and metadata ○ Enumerate jobs and scripts ○ Parse plan info ○ Parse queries
- 29. Use Cases
- 30. #1: Platform Reliability ● SLA Recommendation ● Predict SLA miss ● Instructive notifications ● Anomaly detection
- 31. #2 : Data visibility ● Reinstate key tables ● Deprecate tables ● Schema changes ● Contract no longer held
- 32. #3: Efficiency ● Cost evaluation for entities ○ Storage ○ Compute ○ Usage ● Retention ● Disaster recovery
- 33. Please visit us at jobs.netflix.com Continue your journey... Lineage blog Infra DE team blog
- 34. Netflix @ Strata ● Wednesday ○ Self-service data platform ● Thursday ○ App performance measurement ○ Security detection platform ○ Personalization enhancement 4:40 PM @ 2002 11:50 AM @ 2002 11:50 AM @ 2006 2:40 PM @ 2024
- 35. ???
- 36. Thank you.
- 37. “Provide a complete and accurate data lineage system enabling decision-makers to win moments of truth” Lineage Mission