Dawarehouse como servicio en azure (sqldw)
0 likes423 views
This document discusses Microsoft's data warehousing and analytics solutions including SQL Server, Microsoft Analytics Platform System (APS), and Microsoft Azure SQL Data Warehouse (SQLDW). It provides information on the capabilities and scalability of each solution, noting that SQL Server is ideal for small to medium sized datamarts, APS is for large on-premise data warehouses from 10s of TB to petabytes in size, and SQLDW is for scalable cloud data warehousing from 10s of TB to petabytes. Various topics are summarized around loading data into SQLDW using BCP, SSIS, and PolyBase as well as tuning SQLDW performance.
Dawarehouse como servicio en azure (sqldw)
- 2. #TechSum
- 3. #TechSum [email protected] | www.solidq.com @enriquecatala | www.enriquecatala.com MASTER BI EMPIEZA EN 14 DIAS! http://www.solidq.com/es/masterbi
- 4. #TechSum
- 5. #TechSum Microsoft SQL Server Entorno escalable SMP y NUMA en cualquier hardware OnPremise y Azure IaaS Ideal para datamarts desde pequeños a “medianos” Solo software 10s de TB Microsoft Analytics Platform System (APS) Appliance para entornos datawarehouse de alto rendimiento MPP Solo OnPremise Ideal para alto rendimiento y alta escalabilidad de DW Applicance (SW y HW) 10s de TB – 6 PB (PDW) 24TB – 1.2 PB (Hadoop) Microsoft Azure SQL Data Warehouse (SQLDW) Cloud data warehouse para entornosd de alto rendimiento MPP Solo cloud Ideal para alto rendimiento, alta escalabilidad en cloud Solución completa PaaS 10s de TB – PBs
- 6. #TechSum
- 7. #TechSum
- 9. #TechSum
- 10. #TechSum
- 11. #TechSum Un único cerebro Requiere un motor relacional distribuido
- 14. #TechSum
- 15. #TechSum
- 16. #TechSum Load rate • Records/sec • Network+CPU Scan/Aggregationrate • Records/sec leyendo datos • Operacion compleja • CPU + IO Create Table As Select rate • Velocidad de copia • CPU + IO + NETWORK
- 17. #TechSum DWU Max concurrent queries Concurrency slots allocated DW100 4 4 DW200 8 8 DW300 12 12 DW400 16 16 DW500 20 20 DW600 24 24 DW1000 32 40 DW1200 32 48 DW1500 32 60 DW2000 32 80 DW3000 32 120 DW6000 32 240
- 18. #TechSum
- 19. #TechSum
- 20. #TechSum
- 22. #TechSum
- 23. #TechSum
- 24. #TechSum
- 25. #TechSum
- 26. #TechSum
- 27. #TechSum
- 28. #TechSum
- 29. #TechSum
- 30. #TechSum BCP SSIS Polybase PolyBase BCP SQLBulkCopy/ADF SSIS Load Rate Rate increase as you increase DWU Yes No No No Rate increase as you add concurrent load No Yes Yes Yes FASTEST============>>>>>>>>>>SLOWEST Pero si que se incrementa en Polybase cuando tenemos muchos ficheros por carpeta
- 31. #TechSum
- 32. #TechSum
- 33. #TechSum
- 34. #TechSum NOTA: Todos los tipos de datos tienen workaround, la mayoría valen con varbinary(8000)
- 35. #TechSum ANSI joins on updates ANSI joins on deletes merge statement cursors SELECT..INTO INSERT..EXEC common table expressions recursive common table expressions (CTE) • $partition function • table variables • table value parameters • distributed transactions • commit / rollback work • save transaction • execution contexts (EXECUTE AS) • group by clause with rollup / cube / grouping sets options • nesting levels beyond 8 • updating through views • use of select for variable assignment • no MAX data type for dynamic SQL strings
- 36. #TechSum • E • T • L ETL SERVER DATA WAREHOUSE SERVER EXTRACT TRANSFORM LOAD ELT SERVER DATA WAREHOUSE SERVER EXTRACT TRANSFORM LOAD ELTETL IN-MEMORY TRANSFORMS FEWER CPUS IN-DB MORE CPUS GOOD SORT
- 37. #TechSum DBSource1DBSource1 DBSource2DBSource2 DBSource3DBSource3 SQLDW (PaaS) 5 2 3 Report2 2 4 3 7 1 Report1 Powerpivot SSIS Incremental export Onpremise CPD1 CLOUD (PaaS) Storage (Azure) POLYBASE DBSource1 DBSource2 DBSource3 DB STG BI Database CUBE PreCleanData DWH Consolidation SSIS SSIS SSAS processing SSAS Onpremise CPD1 OnPremise CPD2 Generadas por paquetes CDC
- 38. #TechSum
- 39. #TechSum Pruebas y mediciones • Empezar en 400DTW • Probar con 1000DTW Cambio ETL a ELT • Fuera SSIS • PowerBI/Reporting Patron de subida • Marcas de cambios • OnPrem a SQLDW Carga de datos • Ajustes del modelo
- 40. #TechSum Exportar datos a CSV • SSIS • Automatizacion • BIML Subir CSVs a Azure Storage • AzCopy • PowerShell Cargar CSVs en SQLDW • Create Table from Select As… • Automatizacion • t4dacfx2tsql
- 41. #TechSum
- 42. #TechSum
- 43. #TechSum Bytes_per_row: 250 #_rows: 60,000,000,000 B_to_GB_factor: 1,000,000,000 Distribution_count: 60 Compression_factor: 5 (typical) Partition_count: 36 (monthly over three years) Max_rows_per_rowgroup rows_per_distribution = #_rows / distribution_count rows_per_partition = rows_per_distribution / partition_count max_rowgroups_per_partition = partition_count / max_rows_per_rowgroup Uncompressed_table_size_b = bytes_per_row x # rows Uncompressed_table_size_gb = Uncompressed_table_size_b / B_to_GB_factor Uncompressed_table_distribution_size_gb = uncompressed_table_size_gb/distribution_count Compressed_distribution_size_gb = Uncompressed_table_distribution_size_gb /compression_factor Compressed_partition_size_gb = Compressed_distribution_size_gb / partition_count
- 44. #TechSum Pausa tu SQLDW Ahorrarás dinero Puedes seguir subiendo datos con Polybase DWU €/Hora €/Mes laboral €/Mes completo 100 1,02 € 179,52 € 734,40 € 200 2,04 € 359,04 € 1.468,80 € 300 3,06 € 538,56 € 2.203,20 € 400 4,08 € 718,08 € 2.937,60 € 500 5,10 € 897,60 € 3.672,00 € 600 6,12 € 1.077,12 € 4.406,40 € 1000 10,20 € 1.795,20 € 7.344,00 € 1200 12,24 € 2.154,24 € 8.812,80 € 1500 15,30 € 2.692,80 € 11.016,00 € 2000 20,40 € 3.590,40 € 14.688,00 € 3000 30,60 € 5.385,60 € 22.032,00 € 6000 61,21 € 10.772,96 € 44.071,20 €
- 45. #TechSum Diseña pensando en distribucion de datos Crea estadísticas tras CREATE TABLE Actualiza estadísticas tras carga de datos Particiona tus objetos Comienza por DTW 300
- 46. #TechSum Utiliza BIML Carga mediante Polybase Crea tus objetos mediante T4 Separador complejo $|x
- 47. #TechSum Dinero • No es un problema • Prefiero parar y arrancar BI • Si no tengo solución BI • O no quiero invertir más y quiero la via del hierro Tiempo • Hay mucho dato y necesito saltarme transformacion • Ocasionalmente pido información que necesito rapido
- 49. © 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.