SQL Saturday Redmond 2019 ETL Patterns in the Cloud
Download as PPTX, PDF1 like323 views
This document discusses ETL patterns in the cloud using Azure Data Factory. It covers topics like ETL vs ELT, scaling ETL in the cloud, handling flexible schemas, and using ADF for orchestration. Key points include staging data in low-cost storage before processing, using ADF's integration runtime to process data both on-premises and in the cloud, and building resilient data flows that can handle schema drift.
Ad
More Related Content
What's hot (20)
Similar to SQL Saturday Redmond 2019 ETL Patterns in the Cloud (20)
Ad
More from Mark Kromer (9)
Ad
Recently uploaded (20)
SQL Saturday Redmond 2019 ETL Patterns in the Cloud
- 1. ETL Patterns in the Cloud with Azure Data Factory Mark Kromer Senior Program Manager Microsoft Azure Data Management @kromerbigdata
- 2. ETL Patterns in the Cloud Important factors for success 1. What is ETL? • More than Extract, Transform, Load • Scheduling, Monitoring, Maintenance, Source Control, CI/CD, Operationalize 2. Platform as a Service (ADF) vs. Infrastructure as a Service (IaaS/SSIS) • Self-managed vs. Provider-Managed 3. ELT or ETL? • Difference is primarily highly-parsed semantics • However: In the cloud, common pattern == stage data in low-cost, inexpensive storage 4. Not typically performant to process data in-flight • Particularly crossing boundaries (on-prem, vnets, data centers, regions) 5. Scale is very important in Cloud ETL • Cloud projects assume elastic scale. ETL is not immune to this expectation. 6. Flexible Schema is very important in Cloud ETL 1. Assume “Big Data tenets” aka “data chaos”: Your data sources will change shape, size and volume. Often!
- 3. Azure Data Factory Workflow Pipelines/Control Flow
- 7. ADF Integration Runtime Activity Dispatch/Monitor Data Movement SSIS Package Execution
- 8. Azure Data Factory Service Cloud Apps, Svcs & DataOn Premises Apps & Data UX & SDK
- 9. Azure (US West) Public Internet Border HP Inc (Global) HP Prod Firewall Border HP Hadoop Cluster Integration Fabric On-prem Data Factory (Orchestration micro-service) 443 Storage (Azure) 443 SQL Data Warehouse UAM Server 443 ADF Foo On-prem “IR” Customer 1 Customer 1 firewall border Azure Data Factory “Integration Runtime” deployed on premises for transformation and then moved to cloud
- 10. SSIS in ADF
- 12. on premises in Azure Deployment via SSMS Once connected, you can deploy projects/packages to SSIS PaaS from your local file system/SSIS on premises
- 13. Execution via SSMS You can select some packages to execute on SSIS PaaS
- 14. Monitoring via SSMS on premises in Azure You can see package execution error messages
- 15. Execute SSIS Packaged in ADF Pipeline
- 16. ADF Mapping Data Flows
- 17. What is ADF Mapping Data Flow? Transform Data, At Scale, in the Cloud, Zero-Code Cloud-first, scale-out ELT Code-free dataflow pipelines Serverless scale-out transformation execution engine Maximum Productivity for Data Engineers Does NOT require understanding of Spark / Scala / Python / Java Resilient Data Transformation Flows Built for big data scenarios with unstructured data requirements Operationalize with Data Factory scheduling, control flow and monitoring
- 18. Code-free Data Transformation At Scale Does not require understanding of Spark, Big Data Execution Engines, Clusters, Scala, Python … Focus on building business logic and data transformation Data cleansing Aggregation Data conversions Data prep Data exploration
- 19. ADF Data Flow Workstream Stage Data in Azure (ADLS, Blob, SQL DB/DW) Transform Data in Visual Data Flow Land Data in Azure Staging Area (ADLS, Blob, SQL DB/DW)
- 20. Build your logical data flows adding data transformations in a guided experience
- 21. Microsoft Azure Data Factory Continues to Extend Data Flow Library with a Rich Set of Transformations and Expression Functions
- 22. Debug mode provides row-level context and visible results in inspector pane
- 23. Interactive Expression Builder – Build data transform expressions, not Spark code
- 24. Deep Monitoring Introspection of Data Transformations
- 25. Debug Data Flows with Data Preview and Data Sampling
- 26. Cloud ETL Patterns with ADF
- 28. MODEL & SERVE Azure Analysis ServicesAzure SQL Data Warehouse Power BI Modernize your enterprise data warehouse at scale A Z U R E D A T A F A C T O R Y On-premises data Oracle, SQL, Teradata, fileshares, SAP Cloud data Azure, AWS, GCP SaaS data Salesforce, Workday, Dynamics INGEST STORE PREP & TRAIN Azure Data Factory Azure Blob Storage Azure Databricks Polybase Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure SQL Database and Azure Data Lake to allow customers to tailor the above architecture to meet their unique needs. Orchestrate with Azure Data Factory
- 29. Lift your SQL Server Integration Services (SSIS) packages to Azure On-Premise data sources SQL DB Managed Instance SQL Server VNET Azure Data Factory SSIS Cloud ETL SSIS Integration Runtime Cloud data sources Cloud On-premises Microsoft SQL Server Integration Services
- 30. Author, orchestrate and monitor with Azure Data Factory Hybrid and Multi-Cloud Data Integration Azure Data Factory PaaS Data Integration DATA SCIENCE AND MACHINE LEARNING MODELS ANALYTICAL DASHBOARDS USING POWER BI DATA DRIVEN APPLICATIONS On-Prem SaaS Apps Public Cloud
- 31. Nightly ETL Data Flows - Codeless
- 32. Build Resilient Data Flows with Schema Drift Handling of Flexible Schemas
- 33. Data Engineer derives columns using template expression patterns based on name and type matching. No need to define static field names.
- 34. Sink all incoming fields along with new derived field
- 35. Slowly Changing Dimension Scenario
- 36. Load Star Schema DW Scenario
- 37. Data Lake Data Science Scenario
- 38. Sponsors