Part 3 - Modern Data Warehouse with Azure Synapse
0 likes227 views
Slide deck of the third part of building Modern Data Warehouse using Azure. This session covered Azure Synapse, formerly SQL Data Warehouse. We look at the Azure Synapse Architecture, external files, integration with Azuer Data Factory. The recording of the session is available on YouTube https://www.youtube.com/watch?v=LZlu6_rFzm8&WT.mc_id=DP-MVP-5003170
Ad
More Related Content
What's hot (20)
Similar to Part 3 - Modern Data Warehouse with Azure Synapse (20)
Ad
More from Nilesh Gule (20)
Ad
Recently uploaded (20)
Part 3 - Modern Data Warehouse with Azure Synapse
- 1. Nilesh Gule @nileshgule | www.HandsOnArchitect.com Modern Data Warehouse Using Azure
- 2. $whoami { โnameโ : โNilesh Guleโ, โwebsiteโ : โhttps://www.HandsOnArchitect.com", โgithubโ : โhttps://github.com/NileshGule" โtwitterโ : โ@nileshguleโ, โlinkedinโ : โhttps://www.linkedin.com/in/nileshguleโ, โemailโ : โ[email protected]", โlikesโ : โTechnical Evangelism, Cricketโ, โco-organizerโ : โAzure Singapore UGโ }
- 6. Part 1 - Recap โ ADLS & ADF โข Petabyte scale storage โข Hierarchical namespace โข Hadoop compatible access with ABFS driver ADLS - Main features ADF - Main features โข Cloud ETL service โข Scale-out serverless data integration & data transformation โข Code-free UI โข Monitoring & Management
- 7. Part 2 - Recap โข Collaborative Spark based Analytical service โข Different cluster types (automated / interactive / pool) โข Autoscale based on workloads โข Fine grained access controls Azure Databricks - Main features
- 8. Azure Synapse Limitless analytics service for enterprise data warehousing and Big Data analytics
- 9. Parallelism โข Uses many separate CPUs running in parallel to execute a single program โข Shared Nothing: Each CPU has its own memory and disk (scale-out) โข Segments communicate using high-speed network between nodes MPP - Massively Parallel Processing โข Multiple CPUs used to complete individual processes simultaneously โข All CPUs share the same memory, disks, and network controllers (scale-up) โข All SQL Server implementations up until now have been SMP โข Mostly, the solution is housed on a shared SAN SMP - Symmetric Multiprocessing
- 10. Synapse Architecture https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/massively-parallel-processing-mpp-architecture โข Control Node โข Compute Node โข Data Movement Service (DMS) Components โข Hash โข Round Robin โข Replicate Distributions
- 11. Synapse Data Distributions https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/massively-parallel-processing-mpp-architecture โข Highest query perf for joins & aggregations on large tables โข Rows per distribution varies Hash โข Fastest query performance for small tables Replicated
- 12. ALTER DATABASE ContosoDW MODIFY (service_objective = 'DW1000'); DWU DW100 DW200 DW300 DW400 DW500 DW1000 DW1500 DW2000 DW2500 DW3000 DW5000 DW6000 DW7500 DW10000 DW15000 DW30000
- 13. Azure SQL Data Warehouse Engine Worker1 Azure Storage Blob(s) D12D11 D13 D14 D15 D16 D18D17 D19 D20 D22D21 D23 D24 D25 D26 D28D27 D29 D30 D32D31 D33 D34 D35 D36 D38D37 D39 D40 D42D41 D43 D44 D45 D46 D48D47 D49 D50 D52D51 D53 D54 D55 D56 D58D57 D59 D60 D2D1 D3 D4 D5 D6 D8D7 D9 D10
- 14. Azure SQL Data Warehouse Engine Worker4 Azure Storage Blob(s) Worker1 Worker5 Worker3 Worker2 Worker6 D52D51 D53 D54 D55 D56 D58D57 D59 D60 D12D11 D13 D14 D15 D16 D18D17 D19 D20 D22D21 D23 D24 D25 D26 D28D27 D29 D30 D32D31 D33 D34 D35 D36 D38D37 D39 D40 D42D41 D43 D44 D45 D46 D48D47 D49 D50 D2D1 D3 D4 D5 D6 D8D7 D9 D10
- 15. Azure Databricks โ SQL DW Connectivity
- 16. External Data Sources โข External Data Source โข Hadoop, ADLS โข External File Format โข File types โข Delimited Text, Hive RCFile, Hive ORC file, Parquet โข Data Compression โข Gzip, Snappy โข Field Delimiters โข Date Format โข External Table
- 17. What workloads are NOT suitable? โข High frequency reads and writes. โข Large numbers of singleton selects. โข High volumes of single row inserts. Operational workloads (OLTP) โข Row by row processing needs. โข Incompatible formats (XML). Data Preparations SQL SQL
- 18. What Workloads are Suitable? Store large volumes of data. Consolidate disparate data into a single location. Shape, model, transform and aggregate data. Batch/Micro-batch loads. Perform query analysis across large datasets. Ad-hoc reporting across large data volumes. All using simple SQL constructs. Analytics
- 19. Summary โข MPP Architecture โข Can be paused โข Optimized for analytics workloads โข Supports multiple external file formats โข Works with Polybase Azure Synapse - Main features
- 20. SQL Server & SQL Data Warehouse Differences Azure Synapse Workload Management External Data Source External File Formats External Table SQL Data Warehouse Benchmark
- 21. References โ MS Learn https://docs.microsoft.com/en-us/learn/paths/implement-sql-data-warehouse
- 22. Thank you very much Code with Passion and Strive for Excellence https://www.slideshare.net/nileshgule/presentations https://speakerdeck.com/nileshgule/
- 23. Nilesh Gule ARCHITECT | MICROSOFT MVP โCode with Passion and Strive for Excellenceโ nileshgule @nileshgule Nilesh Gule NileshGule www.handsonarchitect.com
- 24. Q&A