Running cost effective big data workloads with Azure Synapse and ADLS (MS Ignite 2020, OD220)
Download as PPTX, PDF1 like425 views
Presentation by James Baker and myself on Running cost effective big data workloads with Azure Synapse and Azure Datalake Storage (ADLS) at Microsoft Ignite 2020. Covers Modern Data warehouse architecture supported by Azure Synapse, integration benefits with ADLS and some features that reduce cost such as Query Acceleration, integration of Spark and SQL processing with integrated meta data and .NET For Apache Spark support.
Ad
More Related Content
What's hot (20)
Similar to Running cost effective big data workloads with Azure Synapse and ADLS (MS Ignite 2020, OD220) (20)
Ad
More from Michael Rys (20)
Ad
Recently uploaded (20)
Running cost effective big data workloads with Azure Synapse and ADLS (MS Ignite 2020, OD220)
- 1. Running cost effective big data workloads with Azure Synapse and Azure Data Lake Storage James Baker, @BigJamesBigData Michael Rys, @MikeDoesBigData
- 2. Agenda 1. Modernize your big data workloads 2. Cost improvements with Synapse and ADLS 3. Demo 4. .NET for Apache Spark
- 3. Traditional on-prem analytics pipeline Operational database Business/custom apps Operational database Operational database Enterprise data warehouse Data mart Data mart Data mart ETL ETL ETL ETL ETL ETL ETL Reporting Analytics Data mining
- 4. Modern data warehouse Logs (structured) Media (unstructured) Files (unstructured) Business/custom apps (structured) Ingest Prep & train Model & serve Store Azure Data Lake Storage Azure SQL Data Warehouse Azure DatabricksAzure Data Factory Power BI
- 5. Modern data warehouse with Azure Synapse Logs (structured) Media (unstructured) Files (unstructured) Business/custom apps (structured) Azure Synapse Analytics Power BI Store Azure Data Lake Storage
- 6. Modern data warehouse with Azure Synapse Logs (structured) Media (unstructured) Files (unstructured) Business/custom apps (structured) Analytics runtimes SQL Common data estate Shared meta data Unified experience Synapse Studio Store Azure Data Lake Storage Power BI
- 7. Synapse component benefits Existing tooling and skills No requirement for retraining or new tooling to work with familiar T-SQL environments • .NET Support Reducing training costs for big data .NET developers Provisioned model Workload Groups - maximize resource utilization Materialized views and resultset cache - smaller, cheaper clusters Apache Spark Synapse SQL • Hyperspace materialized views Requires smaller, cheaper clusters to achieve the same tasks Serverless model Paying exactly for what you use - no overprovisioning no clusters to monitor and manage - lower maintenance costs
- 8. Cost optimization with Azure Data Lake Storage Disaggregated compute and storage with shared metadata layer Lifecycle management for optimizing TCO Lower compute resources because of high performance
- 9. Azure Synapse and ADLS: Integration benefits Integrated workload monitoring Shared security Query Acceleration Shared metadata Single management portal Data stewards apply security policies only once Data owners define data model once only Smaller, cheaper clusters Reduced monitoring and diagnosis costs Lowers training costs
- 10. Synapse SQL - Serverless Shared Metadata Experience Spark Databases and Tables backed by Parquet become automatically available in: • Synapse SQL serverless • Synapse SQL provisioned • Synapse Pipelines as external tables of the same name. Spark Compute X Auto-Expose Metadata Objects CREATE DATABASE DBS1 CREATE SCHEMA $DBS1 CREATE DATABASE DBS1 CREATE TABLE DBS1.T1 CREATE EXTERNAL TABLE DBS1.dbo.T1 CREATE EXTERNAL TABLE $DBS1.T1 Auto-Expose Metadata Objects Benefits: • No need to run orchestration jobs to move data or meta data between computes • No duplication of data at the storage level SELECT * FROM DBS1.dbo.T1 SELECT * FROM DBS1.T1 SELECT * FROM $DBS1.T1 Synapse Hive Metastore Serverless SQL System Catalog Provisioned Synapse SQL DB Synapse SQL - Provisioned Instance
- 11. Synapse integration with ADLS Query Acceleration Reduces total cost of ownership because analytics frameworks don’t need to parse and load as much data Delivers performance improvements due to less data transferred over network Optimize access to structured data by filtering data directly in the storage service Analytics queries typically require only ~20% of total data read Deeply integrated into Azure Synapse Analytics for improved performance and cost AzureDataLakeStorage Query Acceleration 1 2 5 4 Data 3 Azure Synapse Analytics
- 12. Demo: .NET for Spark and shared metadata experience in Azure Synapse Michael Rys, @MikeDoesBigData
- 13. Demo: .NET for Spark and shared metadata experience in Azure Synapse Analysis with interactive .NET for Spark Notebook Data prep with Spark Scala Twitter CSV files Seamless analysis with Synapse SQL What has Michael been up? Mentions Topics Who was interacting with Michael? Michael @MikeDoesBigData Using Query Acceleration Synapse Shared Meta Data
- 14. ANNOUNCING: .NET for Apache Spark v1.0 is released! First-class C# and F# bindings to Apache Spark, bringing the power of big data analytics to .NET developers Apache Spark 2.4/3.0 Data Frames, Structured Streaming, Delta Lake .NET Standard 2.0 C# and F# ML.NET Performance optimized with Apache Arrow and HW Vectorization First class integration in Azure Synapse: Batch Submission Interactive .NET notebooks Learn more at http://dot.net/Spark
- 15. Call to action Check out sessions: OD217 - Real-time analytics and BI using Azure Synapse Link for Azure Cosmos DB by Euan Garden DB111 - Building real-time enterprise analytics solutions with Azure Synapse Analytics by Saveen Reddy Introduce to Azure Data Lake Storage – https://aka.ms/adls Get started with Azure Synapse – https://aka.ms/azuresynapse Learn more about .NET for Apache Spark - https://dot.net/Spark Leverage Informatica + Microsoft - DW migration offer: https://aka.ms/SynapseInformaticaPOV Connect with us on Twitter: Tweet us: @MikeDoesBigData @BigJamesBigData Tag with: #OD220
Editor's Notes
- #4: Establish the baseline The need to provision for max utilization/max consumption Architectural brittleness in moving data across physical stores
- #5: Pay for consumption model Compute elasticity Data evolves ‘in place’ within ubiquitous storage service
- #6: Encapsulates the MDW pattern within the Synapse service Retain benefits of pay for consumption & ubiquitous store
- #7: Unified experience leveraging heterogenous set of tools/frameworks Shared meta data service means that table definitions do not need to be restated as pipeline flows
- #8: Spark - .NET for Spark is included by default SQL Serverless pay for use only – means no under-utilized clusters running No clusters == reduced maintenance costs Cost control features – caps for usage to avoid cost blowout Provisioned Workload groups provide query isolation with maximum utilization Workload – prioritize queries Materialized Views, Indexing, and Resultset cache are critically important for minimizing IO, less data read, less data cached, less data processed == smaller cheaper clusters
- #10: Talk about the benefits of the tight integration between Synapse & ADLS: Reduced retraining costs due to use of familiar T-SQL environment & single pane of glass Only need to apply metadata once (metastore + access control) Accelerated IO integration with QA - improved performance == improved cost
- #12: Analytics workloads require you to work with huge amounts of data. But the data typically used is 20% of the total data. So you end up processing more data than you should. With Query Acceleration for ADLS, the filtering is done on the storage layer itself, which helps save cost and improve performance. Query acceleration has been used a lot of customers and is generally available in all regions now.