Data architecture for modern enterprise
0 likes217 views
Synopsis: Modern enterprises anticipate business requirements and work proactively to optimise the outcomes. If they don’t renovate or reinvent their data architectures, they lose customers, and market share. So my talk will be in detailing the importance of data architecture, architectural challenges if is not addressed and a case study - the learnings and success story by fixing the issues at the root - at the data storage & access. Target Audience: Principal Software engineers & Architects Key Takeaways: Importance of Modern Data Architecture, PostgreSQL & JSONB I have given a talk @ https://hasgeek.com/rootconf/elasticsearch-users-meetup-hyderabad/
Ad
More Related Content
What's hot (20)
Similar to Data architecture for modern enterprise (20)
Ad
Recently uploaded (20)
Ad
Data architecture for modern enterprise
- 1. Data Architecture for Modern Enterprises Presented @ hasgeek-rootconf-elasticsearch-users-meetup-hyderabad, Nov 2019
- 2. Hyderabad
- 3. AgendaData - The Core Why Data Architecture? Business Use Case Lessons Learnt The Journey & Success Story
- 4. Data - The Domain Films Character ActorDirector Drama Comedy Thriller Patient Doctor Operation Disease Pills Visit Video Health Care Symptom Rom Nurs
- 5. Data - Modeling Techniques 20XX NoSQL 1990s Object Oriented 1970s Relational Navigational 1960s
- 6. The Data HBR's Alarming Survey Results survey participants: 64 c-level technology and business executives representing large corporations such as American Express, Ford Motor, General Electric, General Motors, and Johnson & Johnson 2019 Big Data and AI Executive Survey
- 7. The Data Architecture Data architecture describes how data is collected, stored, transformed, distributed, and consumed - Wikipedia
- 8. Data Architecture - A (Legacy) Sample More Data Sources 3rd Party APIs Databases, DWH & Lakes Different Files NoSQL Application (or) Business Processing Data Consumers More Data Visualisers 3rd Party Apps In House Apps Files Chaos Calls for Renovation/Reinvention
- 9. Message Bus Data Consumers More Data Visualisers ML Tools Applications Microservices More Data Sources 3rd Party APIs Databases, DWH & Lakes Different Files NoSQL Data Architecture - A (Modern) Sample Application (or) Business Processing
- 10. Core Enablers of Modern Business Solid Data Architecture - The Why? IoT AIBig Data Cloud Native ML Robust Data Architecture
- 11. Client Portfolio Business Use Case Through-Channel Marketing Automation (TCMA) SaaS platform ➔ Accelerates Brand sales in Local Market ➔ Shares comprehensive Analytics on Individual Tactics & Campaigns ➔ Helps Brands to adapt to market change based on data driven insights
- 12. The (Domain) Data Business Use Case CampaignsRetailers Target Audience Order Snapshots Brands Orders and so on...
- 13. The Data Analysis Business Use Case The target Audience, the orders, the order snapshots all capture different data points per group of records Chess List ➔ Classic FIDE ➔ Blitz FIDE ➔ Rapid FIDE Football List ➔ Height ➔ Weight ➔ Nationality Wrestling List ➔ Ring Name ➔ Height ➔ Weight ➔ Debut
- 14. Tech Stack Business Use Case
- 15. R1_Orders R1_Snapshots R1_XYZ Indices R1_Audience _1 R1_Audience _2 R1_Audience _3 Indices R1_Audience _4 R1_Audience _5 R1_Audience _N Indices R2_Orders R2_Snapshots R2_XYZ Indices R2_Audience _1 R2_Audience _2 R2_Audience _3 Indices R2_Audience _4 R2_Audience _5 R2_Audience _N Indices R3_Orders R3_Snapshots R3_XYZ Indices R3_Audience _1 R3_Audience _2 R3_Audience _3 Indices R3_Audience _4 R3_Audience _5 R3_Audience _N Indices RN_Orders RN_Snapshots RN_XYZ Indices RN_Audience _1 RN_Audience _2 RN_Audience _3 Indices RN_Audience _4 RN_Audience _5 RN_Audience _N Indices 450K Tables for 4K Customers - Scalable? Chaos.. Calls for Renovation!! The Challenges Business Use Case
- 16. The Reinvention Business Use Case Microservices Netflix has fixed the Scalability Issues by adopting microservices architecture. So, why not us? NoSQL No need to create multiple tables to support dynamic columns instead, let us leverage NoSQL schema-less approach By Client’s Labs Team Is the foundation (data architecture) robust?
- 17. A Business Unit of The solution Business Use Case proposed by Cloud Native architecture with java microservices Phase#2: Reinvention Hybrid Data Architecture - Schema & Schemaless approach Phase#1: Renovation
- 18. Renovation Highlights Business Use Case BRAND LIST FIRST_NAME LAST_NAME CUSTOM_DATA B1 CHESS Vishwanathan Anand {“blitz”: 2786, “rapid”:2737} B1 CHESS Garry Kasparov {“classic”: 2812, “blitz”: 2801, “rapid”:2783} B2 FOOTBALL Christiano Ronaldo {“nationality”: “Portugal”, “height”: “1.87M”} B2 FOOTBALL Lionel Messi {“nationality”: “Argentina”, “height”: “1.7M”} B3 WRESTLING Geetha Phogat {“coach”: “Mahavir Singh”, “height”:”1.62M”} #1 No Multiple Tables & Indices - Dynamic Columns using JSONB & GIN
- 19. #2 Partitioning Renovation Highlights continues... Business Use Case
- 20. A Business Unit of Renovation Benefits Business Use Case proposed byNo Tables & Indices on the fly ➔ Instead Data was Organised for effective Storage & quick Access ➔ Able to accomodate more Brands with the same infrastructure #2: Maximum Scalability ➔ The data architecture issues were fixed ➔ No New Tech Stack introduction, so not much learning curve & additional deployment effort #1: Minimal Disruption
- 21. Renovation Architecture Verification Phase Business Use Case LIST T-SHIRT AUDIENCE 1 S 1,000 2 M 10,000 3 L 100,000 4 XL 1,000,000
- 22. All reads were < 2 seconds even for the 100K audience list The Proof Business Use Case
- 23. Recap The Fundamentals Data is an Asset Rights to get Repaired Organised Consumer Focused Single Source of Truth
- 24. Thank You! Kayalvizhi Noorul Ameen, Principal Architect @ Pramati Technologies Image Credit: unsplash.com & flaticon.com The Case Study - published in Imaginea