From Single Purpose to Multi Purpose Data Lakes - Broadening End Users
2 likes563 views
Watch full webinar here: https://buff.ly/2Mt555e Historically data lakes have been created as centralized physical data storage platform for data scientists to analyze data. But lately the explosion of big data, data privacy rules, departmental restrictions among many other things have made the centralized data repository approach less feasible. In his recent whitepaper, renowned analyst Rick F. Van Der Lans talks about why decentralized multi purpose data lakes are the future of data analysis for a broad range of business users. Please attend this session to learn: • The restrictions of physical single purpose data lakes • How to build a logical multi purpose data lake for business users • The newer use cases that makes multi purpose data lakes a necessity
Ad
More Related Content
What's hot (20)
Similar to From Single Purpose to Multi Purpose Data Lakes - Broadening End Users (20)
Ad
More from Denodo (20)
Ad
Recently uploaded (20)
From Single Purpose to Multi Purpose Data Lakes - Broadening End Users
- 1. DATA VIRTUALIZATION PACKED LUNCH WEBINAR SERIES Sessions Covering Key Data Integration Challenges Solved with Data Virtualization
- 2. From Single Purpose to Multi Purpose Data Lakes - Broadening End Users Michael Dickson Sales Engineer, Denodo Paul Moxon VP Data Architectures & Chief Evangelist, Denodo
- 3. A data lake is a storage repository that holds a vast amount of raw data in its native format. The data structure and requirements are not defined until the data is needed The current needs for sophisticated data-driven intelligence and data science favored this concept for its simplicity and power Hadoop and its ecosystem provided the foundation that data lakes required: vast storage and processing muscle It also favored the concept of ELT vs ETL: load data first, (maybe) Data Lakes 3
- 4. The early data scientists saw Hadoop as their personal supercomputer. Hadoop-based Data Lakes helped democratize access to state of the art supercomputing with off-the- shelf HW (and later cloud) The industry push for BI made Hadoop–based solutions the standard to bring modern analytics to any corporation Data Lakes – A Data Scientist’s Playground 4
- 5. Data Lakes – Not a Perfect World Physical Nature • Based on Replication. Data Lakes require data to be copied to its physical storage • Replication extends development cycles and costs • Not all data is suitable for replication • Real time needs: Cloud and SaaS APIs • Large volumes: existing EDW • Laws and restrictions Single Purpose • Usage of the data lake is often monopolize by data scientists • New data silo. No clear path to share insights with business users • Lacks the governance, security and quality that business users are used to (e.g. in the EDW) 5
- 6. 6 The Rise of Logical Architectures The Evolution of Analytical Architectures Source: Adopt the Logical Data Warehouse Architecture to Meet Your Modern Analytical Needs Gartner April 2018
- 7. Rick Van der Lans, R20 Consultancy Multi‐purpose data lakes are data delivery environments developed to support a broad range of users, from traditional self‐service BI users (e.g. finance, marketing, human resource, transport) to sophisticated data scientists. Multi‐purpose data lakes allow a broader and deeper use of the data lake investment without minimizing the potential value for data science and without making it an inflexible environment.
- 8. The Multipurpose Data Lake with Data Virtualization Logical Nature • Replication is an option, not a necessity • Broaden data access, shorten development times, better insights • Tight integration with big data systems. Fast execution with large data volumes Multi-purpose • Curated access for non-technical users • Better governance and access control • Better ROI for the investment of the lake 8
- 9. 9 The Multipurpose Data Lake with Data Virtualization “A multi-purpose data lake can become an organization’s universal data delivery system” Architecting the Multi-Purpose Data Lake with Data Virtualization , Rick Van der Lans, April 2018
- 10. Single access to all data assets, internal and external: ▪ Physical Data Lake (usually based on SQL-on- Hadoop systems) ▪ Other databases (EDW, ODS, applications, etc.) ▪ SaaS APIs (Salesforce, Google, social media, etc.) ▪ Files (local, S3, Azure, etc.) The Virtual Data Lake – Access to all Data Sources 10
- 11. The physical Data Lake can also be used as Denodo’s cache This allows to quickly load any data accessible by Denodo to the Hadoop cluster Caching becomes an alternative to ingestion ELT processes that preserves lineage and governance Load process based on direct load to HDFS: 1. Creation of the target table in Cache system 2. Generation of Parquet files (in chunks) with Snappy compression in the local machine 3. Upload in parallel of Parquet files to HDFS The Virtual Data Lake – Ingesting and Caching 11
- 12. Denodo optimizer provides native integration with MPP systems to provide one extra key capability: Query Acceleration Denodo can move, on demand, processing to the MPP during execution of a query • Parallel power for calculations in the virtual layer • Avoids slow processing in-disk when processing buffers don’t fit into Denodo’s memory (swapped data) The Virtual Data Lake – Using the Lake Processing Engine 12
- 13. 13 The Virtual Data Lake – Putting the Pieces Together 2M rows (sales by customer) Current Sales 68 M rows 1. Partial Aggregation push down Maximizes source processing dramatically Reduces network traffic 3. On-demand data transfer Denodo automatically generates and upload Parquet files 4. Integration with local data The engine detects when data is cached or comes from a local table already in the MPP 2. Integrated with Cost Based Optimizer Based on data volume estimation and the cost of these particular operations, the CBO can decide to move all or part of the execution tree to the MPP 5. Fast parallel execution Support for Spark, Presto and Impala for fast analytical processing in inexpensive Hadoop-based solutions Hist. Sales 220 M rows Customer 2 M rows (Cached) join group by ZIP System Execution Time Optimization Techniques Others ~ 10 min Simple federation No MPP 43 sec Aggregation push-down With MPP 11 sec Aggregation push-down + MPP integration (Impala 8 nodes) group by Customer ID
- 14. ▪ A Virtual Data Lake improves decision making and shortens development cycles • Surfaces all company data from multiple repositories without the need to replicate all data into the lake • Eliminates data silos: allows for on-demand combination of data from multiple sources ▪ A Virtual Data Lake broadens adoption of the lake and improves its ROI • Improves governance and metadata management to avoid “data swamps” • Allows controlled access to the lake to non-technical users ▪ A Virtual Data Lake offer performance for the Big Data World • Leverages the processing power of the existing cluster controlled by Denodo’s optimizer The Virtual Data Lake - Conclusions 14
- 15. 15 Gartner, Adopt the Logical Data Warehouse Architecture to Meet Your Modern Analytical Needs, May 2018 When designed properly, DV can speed data integration, lower data latency, offer flexibility and reuse, and reduce data sprawl across dispersed data sources. Due to its many benefits, DV is often the first step for organizations evolving a traditional, repository- style data warehouse into a Logical Architecture”
- 16. Product Demonstration A Multipurpose Data Lake with Data Virtualization 16 Sales Engineer, Denodo Michael Dickson
- 17. 17 Demo Architecture What’s the impact of a new marketing campaign for each country? ▪ Historical sales data offloaded to Hadoop cluster for cheaper storage ▪ Marketing campaigns managed in an external cloud app ▪ Country is part of the customer details table, stored in the DW Sources Combine, Transfor m & Integrate Consume Base View Source Abstraction join group by state join Sales Campaign Customer
- 18. Demo 18
- 19. Key Takeaways 19
- 20. Key Takeaways 20 FIRST Takeaway Hadoop-based Data Lakes are the standard approach to modern analytics within most organizations SECOND Takeaway Physical Data Lakes introduce many complexities (replication, synchronization, governance, etc.) that restrict their use THIRD Takeaway Logical Data Lakes allow users to access data from all sources – internal and external – to grow value of Data Lake approach FOURTH Takeaway Data Virtualization creates ‘multipurpose’ Data Lakes for all kinds of users – data scientists and business users FIFTH Takeaway Data Virtualization introduces governance and access controls to the Data Lake without impeding the ‘power users'
- 21. Q&A
- 22. Next steps Download Denodo Express: www.denodoexpress.com Access Denodo Platform in the Cloud! 30 day FREE trial available! Denodo for Azure: www.denodo.com/TrialAzure/PackedLunch Denodo for AWS: www.denodo.com/TrialAWS/PackedLunch
- 23. Next session The Role of Data Virtualization in an API Economy Thursday, September 20, 2018 | 11:00 AM PT / 2:00 PM ET Paul Moxon VP Data Architectures & Chief Evangelist, Denodo
- 24. Thank you! © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.