Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
13 likes4,321 views
The document discusses using a data lake approach with EMC Isilon storage to address various business use cases. It describes how the solution provides shared storage for multiple workloads through multi-protocol support, enables data protection and isolation of client data, and allows testing applications across Hadoop distributions through a common platform. Examples are given of how this approach supports an enterprise data hub, data warehouse offloading, data integration, and enrichment services.
Ad
More Related Content
What's hot (19)
Similar to Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016 (20)
Ad
More from StampedeCon (20)
Ad
Recently uploaded (20)
Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
- 1. © Copyright 2016 EMC Corporation. All rights reserved. EMC EMERGING TECHNOLOGIES ROBERT HOUT - ADVISORY SYSTEMS ENGINEER @rob_hout ACCELERATING ANALYTICS VALUE WITH A DATA LAKE STAMPEDECON 2016
- 2. What is a Data Lake?
- 3. CONNECTED PEOPLE 2.3B 7B 2015 2020 CONNECTED DEVICES 4.9B 30B 2015 2020 DATA ON PLANET 8ZB 44ZB 2015 2020 2 0 2 0 : A N E W D I G I T A L W O R L D 3X 6X 5X
- 4. Why Ingest All The Data?
- 5. 69% 83% Source: “The Business of Data” and Economist Intelligence Report, Published Jan 2016 Every Organization Can Gain Insights 60% Generating revenue from data Starting new BU developing data-related products / services Used data to make existing products / services more profitable Every Organization is A Data Organization
- 6. The Data Lake: Bringing Compute to Data EDWs Marts Storage Search Servers Documents Archives ERP, CRM, RDBMS, Machines Files, Images, Video, Logs, Clickstreams External Data Sources Mul$-workload analy$c pla1orm • Bring applicaDons to data • Combine different workloads on common data (i.e. SQL + Search) • True BI agility 4 1 2 1 34 Ac$ve archive • Full fidelity original data • Indefinite Dme, any source • Lowest cost storage 1 Data management, transforma$ons • One source of data for all analyDcs • Persisted state of transformed data • Significantly faster & cheaper 2 Self-service exploratory BI • Simple search + BI tools • “Schema on read” agility • Reduce BI user backlog requests 3
- 7. TO SUCCEED, SIMPLIFY TECHNOLOGY SO YOU CAN SHIFT FOCUS TO BUSINESS OUTCOMES KEY CAPABILITIES TO LOOK FOR IN A COMPREHENSIVE BIG DATA SOLUTION INGEST Capture data from a wide range of sources, traditional and new STORE Store everything in one environment for cross data analysis ANALYZE Use advanced algorithms to discover new, predictive patterns SURFACE Share insights with business domain experts ACT Build data-driven applications to meet business needs
- 8. © Copyright 2014 EMC Corporation. All rights reserved. Why a Data Lake It delivers comprehensive data services not a point solution • Our traditional IT customers solve the most pressing issues first, e.g. building a physical Hadoop cluster • Customers are very good at building parts of a data lake that don’t always align to one another • Customers struggle with integrating, managing, and deploying the various platforms needed for business analytics • Customers have little or no overall data governance, but they need it in order to establish a fully functional data lake
- 9. The Data lake high-level Vision • Business-led, cross-functional, methodology focused on short, iterative release cycles • Functional distinction between Data Preparation (IT) and Data Usage (Business) • Enabling on-demand services - BI and Analytics sandboxes, tools, and data Self-Service BI and Analytics • The provisioning of data and services to the business independent of data end usage • A key foundation for of Self-Service BI (Data Preparation) • Services can include publication, profiling, archiving, metadata, alerts, and notifications Data as a Service • Alternative to traditional data warehousing focused on agility, flexibility and time to value • Land data ‘as-is’ and transform on demand (‘schema on read’) • Scale out architecture that is adaptive to business cost/performance constraints Business Data Lake Process Technology DataGovernance
- 10. • Combine different data sources • Minimize data movement • Leverage the Apache ecosystem • Evolve seamlessly • Serve the Enterprise Data Lake implementation strategy needs to… Production Data Web Logs Public Sales Billing CRM SCM Social Media Location Click Streams Sensor Data DATA LAKE
- 11. Security Business Continuity Compliance Tools & Apps Business Units Data Migration PRODUCTION HADOOP HAS SEVERAL CHALLENGES Scalability
- 12. System Availability Uptime Downtime (per year) 99.999% (AKA 5 nines) 5.26 minutes 99.99% (AKA 4 nines) 52.6 minutes 99.5% 1.83 days 99% (AKA 2 nines) 7.30 days 95% 18.25 days What is your Data Warehouses’ uptime SLA? What is your Hadoop uptime SLA? Why are they different?
- 13. • Virtualization becoming more common • Enterprise data management, protection, security • SQL on Hadoop the norm • Spark exploding – Generally Lambda architecture, not Spark vs. M/R • Non-HDFS App Data Integration – ELK, MongoDB, Cassandra.. • High performance/ACID/Mem DBs with HDFS Backend • IoT data collection considerations (HWX Onyara/NiFi) APACHE Ecosystem Trends
- 14. Traditional Hadoop For The Data Lake? Direct-attached storage Stand-alone Servers Single purpose All commodity environment Traditional Hadoop Efficiency, Agility, SLAs Rapid deployment Purpose Built Silos Operational Complexity Enterprise Challenges Reintroduces challenges that Enterprise IT solved years ago
- 15. Hadoop HAS MULTIPLE WORKLOADS “One size fits all” approach to Hadoop Infrastructure does not scale for diverse production workloads Hadoop Archive Spark HBase SQL-on- Hadoop Hive/Tez MapReduce Geo-Dist Hadoop
- 16. COLLECT, STORE, ANALYZE & USE Traditional and Emerging Sources Social Networks, User Generated Content Public records Location DataInternet Of Things Emerging Enterprise File Data Machine Data Traditional Video Archive
- 17. COLLECT, STORE, ANALYZE & USE Traditional and Emerging Sources EmergingTraditional DAS CLOUD OBJECTTAPE SAN NAS
- 18. Isilon Scale-Out Data Lake 18 Data Silos vs Consolidated Data Lake
- 19. • One instance of the file services all dependent workloads simultaneo usly FILE 19 FILE EMC Isilon Next-Gen Access Methods
- 20. • An access zone is: – A way to carve the cluster into smaller clusters – A way to control access based on individual authentication – OneFS’s Multi-Tenancy solution NFS, SMB, HDFS and OpenStack Swift Access Zones Chez NFSAccess Zone-1 System Zone Access Zone-2 Kerberos-1 Domain Controller-2 LDAP-1 NIS - 1 Group Database - 1 Kerberos-2 Domain Controller-1 Group Database - 2
- 21. Data Sharing Across Access Zones • Same files can be accessed by different access zone clients • Best for: – multi-group collaboration w/ untrusted Active Directories – multi-group data access governed by IP subnet – Hadoop analytics over multi access zone data • Uniquely solve collaboration challenge; saves time and money
- 22. DATA LAKE (HADOOP)RDBMS MACHINE IOT STATISTICAL MODELING/NLP VISUALIZATION TRANSFORM BI ORGANIZE MANAGE/ CATALOG DATA WAREHOUSESTREAM CEP NEAR REAL-TIME MODELS MAY TAKE HOUR OR DAYS QUERIES MAY RETURN IN SECONDS OR MINUTES SECONDS SEARCH/INDEX ENTERPRISE LOG ANALYSIS APPLICATIONS 3rd PARTY EMAIL SOCIAL MEDIA SQL ON HADOOP THE BIG DATA LANDSCAPE
- 23. DATA LAKE (HADOOP)RDBMS MACHINE IOT STATISTICAL MODELING/NLP VISUALIZATION TRANSFORM BI ORGANIZE MANAGE/ CATALOG DATA WAREHOUSESTREAM CEP NEAR REAL-TIME MODELS MAY TAKE HOUR OR DAYS QUERIES MAY RETURN IN SECONDS OR MINUTES SECONDS SEARCH/INDEX ENTERPRISE LOG ANALYSIS APPLICATIONS 3rd PARTY EMAIL SOCIAL MEDIA SQL ON HADOOP THE BIG DATA LANDSCAPE
- 24. A Next Gen Data Lake Architecture Clickstream Web & Social Geoloca$on Sensor & Machine Server Logs EXISTINGSOURCES ERP CRM Commodity Compute DATA SERVICES OPERATIONAL SERVICES Hadoop Pla1orm HADOOP CORE Business Analytics Business Analytics Visualization & Dashboards Visualization & Dashboards IT Applications NEWSOURCES 2 3 1 Data Marts Data Management ETL/ELT OFFLOAD ACTIVE ARCHIVE ENRICH WITH NEW DATA TYPES MULTI-PROTOCOL ACCESS ENTERPRISE-GRADE DATA MANAGEMENT 5 NFS, SMB, HTTP, Swift 1 2 3 4 5Isilon 4 New Data Flow Current Data Flow Legend OFFLOAD
- 25. © Copyright 2014 EMC Corporation. All rights reserved. The Data Lake Vision Storage Layer Data Store Manager 3rd Party INGEST MANAGER STREAM Exploratory Analytics Isilon XtremIO ECS/ViPR DSSD DATA GOVERNOR SECURITY INDEXING CATALOGING POLICY Modeling Correlations SQL NSQL BATCH Interactive Analytics Aggregates OLAP SQL NSQL Realtime Analytics Modeling Scoring SQL NSQL In MEM Shared Store(s) Private Store(s) FILE COLUMN DB RELATIONAL DB GRAPH DB KEY VALUE DOCUMENT LOGS FILE BATCH SQL ETL MARKETPLACE MANAGERDATA SERVICES PORTAL VNX APPLICATIONS USERS Analytics Platform ManagerVMware Openstack Docker Evo:Rails
- 26. © Copyright 2014 EMC Corporation. All rights reserved. Ingest Manager Rapid collection of data from unlimited sources Application Services Portal User Services Portal Application Platform Manager Data Ingest Manager Ingest Application Provisioning Ingest Management and Control Catalog Connector Locality Manager Indexing Connector Security Manager Data Governor
- 27. © Copyright 2014 EMC Corporation. All rights reserved. The Data Governor Enabling comprehensive data management Data Catalog Management Security Management Security and Roles LDAP AD BUILT-IN Policy Management Data Types Shared Private Data Sources Consumer Access Rights Compliant Data Sets Encryption/Location Reqs. Lineage Requirements Index Management Licensing Resource Policies Usage Limits Index Management Index Type Index Usage Indexing Resources Index Engine(s) Data Catalog Types Public Private Catalog Type Catalog Usage Catalog Resources Catalog Engine(s) Catalog Security and Roles Catalog Operations Collection Scavenging
- 28. © Copyright 2014 EMC Corporation. All rights reserved. Data Store Manager Manage and provision storage for a variety of uses Application Platform Manager Data Store Manager Storage Manager Shared Stores Data GovernorPrivate Stores Compliant Stores Temporary Stores 3rd PartyIsilon XtremIO ECS/ViPR DSSDVNX Storage Configuration and Provisioning Manager
- 29. © Copyright 2014 EMC Corporation. All rights reserved. Application Provisioning Application Platform Manager Rapidly and seamlessly deploy applications and resources Application Services Portal User Services Portal Application Platform ManagerApplication Platform Manager Application Provisioning Compute Resource Provisioning Platform Provisioning VM Workflows Provisioning Rules Networking Optimizations Application Deployment Management Package Manager App Store Manager Data Store Manager Data Governor
- 30. © Copyright 2014 EMC Corporation. All rights reserved. An example analytic workflow From idea to action using the BDL Application Provisioning User Services Portal Application Platform Manager Platform Provisioning Data Store Manager Data Governor Data Catalog Management Security Management Policy Management Index Management Optimization Engine Recommendation Engine
- 32. THE DATA LAKE One Customer’s Journey with Hadoop
- 33. Use Cases & Requirements • As we evaluated business use cases that would support it was determined that we had a variety of workloads with different impacts to the platform Use Cases • Enterprise Data Hub that can consolidate disparate data sources to a common platform (i.e. data types) • Migrate Enterprise Data Warehouse (EDW) transient data to lower cost storage platform • Enable data enrichment services to enable in-record validation, data standardization and analytic processing • Integrate and provision data to target systems using Hadoop ecosystem components (i.e. Pig, Hive) Requirements • Ensure that the platform meets both availability and recoverability targets • Align technology to internal skills and competencies • Enable existing systems to interoperate with the platform using native protocols or services • Ability to test and certify commercial products via a multi-distribution environment • Enable co-resident processing of products to optimize the use of deployed infrastructure • Ability to provide data protection and isolation of client data within a single instance of the platform (i.e. sub-tenancy
- 34. Solution Approach Support for a variety of acquisition channels • 3 Common method for data types and formats Orchestration framework that manages all job execution Includes capabilities around data catalog, file validation, and schema evolution Data integration and provisioning framework Support for relational stores and exploration tools
- 35. Platform Approach As we better defined and understood use cases and requirements, it led us down a different path from a platform perspective Data Warehouse Offload Data Integration Enterprise Data Hub Enrichment Validation and Quality ü The ability to independently scale storage and compute ü Provide data protection for critical business information ü Support backup and disaster recovery ü Centrally managed via intuitive user interface ü Leverage existing assets deployed in the enterprise
- 36. Example: Multi-protocol Support • 3 One of our deployed use cases is multi-protocol support. This enables us to leverage existing assets and talent in the enterprise but can still leverage the compute paradigm of Hadoop
- 37. Example: Multi-distribution Support • 37 Our organization sells a number of products to the market. Many of these deployments are on-premise due to concerns around data privacy or control, data transfer considerations, etc. To support this need a multi-distribution platform was needed that could be used for product certification across similar data sets
- 38. The Isilon Advantage for Hadoop In-place analytics • No data ingest necessary, Isilon provides shared multi-protocol access • Native integration speeds time to insight Enterprise data protection • Fast snapshots, backup, and data recovery • Simple, efficient data replication for disaster recovery Lower costs • Eliminates the need for dedicated Hadoop infrastructure • Eliminates 3x mirroring for data protection • Much more efficient than DAS-based approach Increase flexibility • Simultaneous support for any Apache-compliant Hadoop distribution • Collaborative engineering efforts with Cloudera, Hortonworks, and Pivotal • Ambari integration for management, monitoring, and provisioning Scale-out storage with native Hadoop integration