Building a Logical Data Fabric using Data Virtualization (ASEAN)
0 likes376 views
The document provides an overview of a Lunch & Learn session by Denodo on building a logical data fabric using data virtualization, featuring presentations on data lakes, warehouses, and the importance of integrating data across platforms. It discusses key concepts such as the definitions of data lakes and warehouses, the necessity of both systems, and the role of architects and data scientists in data management. Additionally, the document highlights a case study of Autodesk's transition to a subscription-based model through the implementation of a logical data warehouse, emphasizing performance, data discovery, and analytics capabilities.
![6
What is a Data Warehouse ?
In computing, a data warehouse (DW or DWH), also known as an enterprise
data warehouse (EDW), is a system used for reporting and data analysis and
is considered a core component of business intelligence.[1] DWs are central
repositories of integrated data from one or more disparate sources. They
store current and historical data in one single place[2] that are used for
creating analytical reports for workers throughout the enterprise.[3]
The data stored in the warehouse is uploaded from the operational
systems (such as marketing or sales). The data may pass through
an operational data store and may require data cleansing[2] for additional
operations to ensure data quality before it is used in the DW for reporting.
Source: https://en.wikipedia.org/wiki/Data_warehouse](https://image.slidesharecdn.com/denodoaseandllnov2021-buildingalogicaldatafabricusingdatavirtualization-211123171142/85/Building-a-Logical-Data-Fabric-using-Data-Virtualization-ASEAN-6-320.jpg)
Building a Logical Data Fabric using Data Virtualization (ASEAN)
- 1. DENODO LUNCH & LEARN 23 NOVEMBER BUILDING A LOGICAL DATA FABRIC USING DATA VIRTUALIZATION
- 2. Presenters for this Session Chris Day Director, APAC Sales Engineering, Denodo Regional Vice President, Sales, ASEAN & Korea, Denodo Elaine Chan
- 3. Agenda 1. Data Lakes and Data Warehouses 2. Observations From the Recent TDWI Report 3. What is a Logical Data Fabric ? (Deep Dive) 4. How Does This Apply to a Data Warehouse/Data Lake? 5. Customer Case Study - Autodesk 6. Conclusions 7. Product Demo 8. Q & A and Next Steps
- 4. Building A Logical Data Fabric Using Data Virtualization Regional Vice President, Sales, ASEAN & Korea, Denodo Elaine Chan
- 6. 6 What is a Data Warehouse ? In computing, a data warehouse (DW or DWH), also known as an enterprise data warehouse (EDW), is a system used for reporting and data analysis and is considered a core component of business intelligence.[1] DWs are central repositories of integrated data from one or more disparate sources. They store current and historical data in one single place[2] that are used for creating analytical reports for workers throughout the enterprise.[3] The data stored in the warehouse is uploaded from the operational systems (such as marketing or sales). The data may pass through an operational data store and may require data cleansing[2] for additional operations to ensure data quality before it is used in the DW for reporting. Source: https://en.wikipedia.org/wiki/Data_warehouse
- 7. 7 Etymology of “Data Lake” Pentaho’s CTO James Dixon is credited with coining the term "data lake". He described it in his blog in 2010: “If you think of a data mart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples.” Source: https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/
- 8. 8 Can We Place All Data Into a Single Cloud System? Taking advantage of the current shift to the cloud, couldn't we simply consolidate all data in a single system, like a data lake or a “lakehouse”? § They are (relatively) cheap and scale out well for large data volumes § Is that realistic? § Is that possible?
- 9. 9 Do I Need Both a Data Lake and a Data Warehouse? Data Warehouses: Typically contain structured data and more often than not are on premise • Top use case BI and Analytics Data Lakes: Often contain unstructured data and typically cloud based. • Top use case Data Science 2021 TDWI report Building the Unified Data Warehouse and Data Lake BEST PRACTICES REPORT By Fern Halper, Ph.D., and James Kobielus
- 10. 10 Do I Really Need Both a Data Lake and a Data Warehouse? 2021 TDWI report Building the Unified Data Warehouse and Data Lake BEST PRACTICES REPORT By Fern Halper, Ph.D., and James Kobielus
- 11. 11 Why? 2021 TDWI report Building the Unified Data Warehouse and Data Lake BEST PRACTICES REPORT By Fern Halper, Ph.D., and James Kobielus
- 12. 12 Data lakes were born to efficiently address the challenge of cost reduction: Data lakes allow for cheap, efficient storage of very large amounts of data. Cloud implementation simplified the complexity of managing a large data lake.
- 13. 13 …Data lakes lack semantic consistency and governed metadata. Meeting the needs of wider audiences require curated repositories with governance, semantic consistency and access controls.”
- 14. 14 So How Are Organizations Going About It? 2021 TDWI report Building the Unified Data Warehouse and Data Lake BEST PRACTICES REPORT By Fern Halper, Ph.D., and James Kobielus
- 15. 15 Final Observations From the Report Architects own the overall design. It is no surprise that architects are the top contributors to the design of the data warehousing environment. This includes data warehouse architects (49%), enterprise architects (43%), and IT architects (21%). Data scientists (56%) are the top contributor of various components to the unified environment. 2021 TDWI report Building the Unified Data Warehouse and Data Lake BEST PRACTICES REPORT By Fern Halper, Ph.D., and James Kobielus
- 16. What is a Logical Data Fabric?
- 17. 17 A data fabric is an architecture pattern that informs and automates the design, integration and deployment of data objects regardless of deployment platforms and architectural approaches. It utilizes continuous analytics and AI/ML over all metadata assets to provide actionable insights and recommendations on data management and integration design and deployment patterns. This results in faster, informed and, in some cases, completely automated data access and sharing. Data Fabric Definition
- 18. 18 Data Fabric A data fabric is an architecture pattern that informs and automates the design, integration and deployment of data objects regardless of deployment platforms and architectural approaches § It utilizes continuous analytics and AI/ML over all metadata assets to provide actionable insights and recommendations on data management and integration design and deployment patterns. § This results in faster, informed and, in some cases, completely automated data access and sharing § Strongly supported by both Gartner and Forrester Data Fabric Net Compounds Customers Products Claims RDBMS/OLTP Traditional Analytics/BI Data Lakes Cloud Data Stores Apps and Document Repositories Flat Files Third Party Legacy Mart Data Warehouse Mart ETL ETL XML • JSON • PDF DOC • WEB
- 19. 19 Logical Data Fabric Demystifying the Data Fabric, September 2020 The core of the matter is being able to consolidate many diverse data sources in an efficient manner by allowing trusted data to be delivered from all relevant data sources to all relevant data consumers through one common layer.
- 20. 20 Logical Data Fabric • Data Abstraction: decoupling applications/data usage from data sources • Data Integration without replication or relocation of physical data • Easy Access to Any Data, high performant and real-time/ right- time • Data Catalog for self-service data services and easy discovery • Unified metadata, security & governance across all data assets • Data Delivery in any format with intelligent query optimization that leverages new and existing physical data platforms A logical data layer – a “logical data fabric” – that provides high-performant, real-time, and secure access to integrated business views of disparate data across the enterprise
- 21. How Does This Apply to a Data Warehouse/Data Lake ?
- 22. 22 Logical Data Fabric Reference Architecture LOGICAL DATA FABRIC
- 23. 23 Cloud Applications Salesforce.com AWS Ecosystem AWS Data Sources Amazon Aurora Amazon Dynamo DB Amazon Redshift Amazon Athena Amazon S3 Data Consumers Amazon QuickSight OnPrem Data Sources MS Excel Other Applications SAP Data Sources Example: Denodo in Multiple Locations Azure Ecosystem Denodo Data Virtualization Data Catalog SQL API Denodo Data Virtualization SQL API Azure Data Sources Azure Synapse Azure Delta Lake
- 25. 25 v Autodesk Overview • Founded 1982 (NASDAQ: ASDK) • Annual revenues (FY 2018) $2.06B § Over 8,800 employees • 3D modeling and animation software § Flagship product is AutoCAD • Market sectors: § Architecture, Engineering, and Construction § Manufacturing § Media and Entertainment § Recently started 3D Printing offerings
- 26. 26 v Business Drivers for Change • Software consumption model is changing § Perpetual licenses to subscriptions § User want more flexibility in how they use software • Autodesk needed to transition to subscription pricing § 2016 – some products will be subscription only • Lifetime revenue higher with subscriptions § Over 3-5 years, subscriptions = more revenue • Changing a licensing model is disruptive
- 27. 27 v Technology Challenges • Current ‘traditional’ BI/EDW architecture not designed for data streams from online apps § Weblogs, Clickstreams, Cloud/Desktop apps, etc. • Existing infrastructure can’t simply ‘go away’ § Regulatory reporting (e.g. SEC) § Existing ‘perpetual’ customers • ‘Subscription’ infrastructure work in parallel § Extend and enhance existing systems § With single access point to all data • Solution – ‘Logical Data Warehouse/Fabric’
- 32. 32 v Autodesk Successfully Changes Their Revenue Model and Transforms Business § Autodesk was changing their business revenue model from a conventional perpetual license model to subscription- based license model. § Inability to deliver high quality data in a timely manner to business stakeholders. § Evolution from traditional operational data warehouse to contemporary logical data warehouse deemed necessary for faster speed. § Successfully transitioned to subscription- based licensing. § For the first time, Autodesk can do single point security enforcement and have uniform data environment for access. § General purpose platform to deliver data through logical data warehouse. § Denodo Abstraction Layer helps live invoicing with SAP. § Data virtualization enabled a culture of “see before you build”. 32 Autodesk, Inc. is an American multinational software corporation that makes software for the architecture, engineering, construction, manufacturing, media, and entertainment industries. Case Study
- 33. Conclusions
- 34. 34 A Logical Data Fabric § Pillar 1 — Integrates data across multi-cloud environments § Pillar 2 - Automates manual tasks using augmented intelligence § Pillar 3 - Boosts performance of analytics with rapid data delivery § Pillar 4 - Supports data discovery and data science initiatives § Pillar 5 - Analyzes across data at rest and data in motion § Pillar 6 - Catalogs all data for discovery, lineage, and associations TDWI Checklist Report: Six Critical Capabilities of a Logical Data Fabric (May 2020) By Fern Halper and David Loshin - denodo.link/tdwi5
- 35. Product Demonstration Director, APAC Sales Engineering, Denodo Chris Day
- 36. Q&A
- 37. Next Steps
- 39. Featuring Leading Industry Experts Angel Vina Founder & CEO Alberto Pan Executive VP & CTO Ravi Shankar Senior VP & CMO David Loshin President of Knowledge Integrity Terry Moon Enterprise Information Architect Logical Data Fabric: The Future of Data Management and Analytics Michele Goetz VP & Principal Analyst denodo.link/DF2111 AVAILABLE ON DEMAND
- 40. Thanks! www.denodo.com [email protected] © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.