Columnstore improvements in SQL Server 2016
1 like580 views
Niko Neugebauer gave a presentation on the columnstore improvements in SQL Server 2016. Some of the key improvements discussed include hybrid transactional/analytical processing (HTAP), new T-SQL syntax for defining columnstore indexes, high availability features like readable secondaries, improved data loading and batch processing performance, new maintenance features, and expanded monitoring capabilities. The presentation provided examples and demonstrations of many of these new columnstore features in SQL Server 2016.
![New T-SQL for Disk-Based Columnstore
CREATE TABLE [dbo].[FactOnlineSales_CCI](
[OnlineSalesKey] [int] NOT NULL,
[StoreKey] [int] NOT NULL,
[ProductKey] [int] NOT NULL,
[PromotionKey] [int] NOT NULL,
[CurrencyKey] [int] NOT NULL,
[CustomerKey] [int] NOT NULL,
INDEX PK_FactOnlineSales_CCI CLUSTERED COLUMNSTORE
);
CREATE TABLE [dbo].[FactOnlineSales_NCCI](
[OnlineSalesKey] [int] NOT NULL,
[StoreKey] [int] NOT NULL,
[ProductKey] [int] NOT NULL,
[PromotionKey] [int] NOT NULL,
[CurrencyKey] [int] NOT NULL,
[CustomerKey] [int] NOT NULL,
INDEX PK_FactOnlineSales_CCI NONCLUSTERED COLUMNSTORE (OnlineSalesKey, CustomerKey)
);](https://image.slidesharecdn.com/columnstoreimprovementsinsqlserver2016-171128173628/85/Columnstore-improvements-in-SQL-Server-2016-35-320.jpg)
![New T-SQL for In-Memory Columnstore
CREATE TABLE [dbo].[FactOnlineSales_Hekaton](
[OnlineSalesKey] [int] NOT NULL,
[StoreKey] [int] NOT NULL,
[ProductKey] [int] NOT NULL,
[PromotionKey] [int] NOT NULL,
[CurrencyKey] [int] NOT NULL,
[CustomerKey] [int] NOT NULL,
Constraint PK_FactOnlineSales_Hekaton PRIMARY KEY NONCLUSTERED ([OnlineSalesKey]),
INDEX CCI_FactOnlineSales_Hekaton
CLUSTERED COLUMNSTORE
) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);
ALTER TABLE dbo. FactOnlineSales_Hekaton
ADD INDEX CCI_FactOnlineSales_Hekaton
CLUSTERED COLUMNSTORE;](https://image.slidesharecdn.com/columnstoreimprovementsinsqlserver2016-171128173628/85/Columnstore-improvements-in-SQL-Server-2016-36-320.jpg)
Columnstore improvements in SQL Server 2016
- 1. Columnstore Improvements in SQL Server 2016 Niko Neugebauer, OH22
- 3. Explore Everything PASS Has to Offer FREE ONLINE WEBINAR EVENTS FREE 1-DAY LOCAL TRAINING EVENTS LOCAL USER GROUPS AROUND THE WORLD ONLINE SPECIAL INTEREST USER GROUPS BUSINESS ANALYTICS TRAINING VOLUNTEERING OPPORTUNITIES PASS COMMUNITY NEWSLETTER BA INSIGHTS NEWSLETTERFREE ONLINE RESOURCES
- 4. Session Evaluations ways to access Go to passSummit.com Download the GuideBook App and search: PASS Summit 2016 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide Submit by 5pm Friday November 6th to WIN prizes Your feedback is important and valuable. 3
- 5. Niko Neugebauer Consultant, OH22 • Microsoft Data Platform Professional • OH22 (http://www.oh22.net) • Data Platform MVP • Founder of a couple of Portuguese PASS Chapters • Creator of CISL – Free & Open Source Columnstore Indexes Script Library (https://github.com/NikoNeugebauer/CISL) Place your photo here /WebCaravela @NikoNeugebauer www.nikoport.com
- 6. Our Today’s Columnstore Agenda p. 1 • HTAP aka Operational Analytics • Data Warehousing (CCI) • T-SQL • High Availability • Batch Mode Changes • Data Loading (+Minimal Logging) • Performance (Batch, SSIS)
- 7. Our Today’s Columnstore Agenda p. 2 • Memory Managment • Change Tracking • Maintenance • Monitoring Improvements (DMV, Extended Events, Performance Counters) • Configuration Improvement (Test it!) • Indexed Views • New Trace Flags • Upcoming Attraction
- 10. What is it all about: • Predictive Analytics, Prescriptive Analytics, Business Analytics, Machine Learning, Data Mining But technically it all goes down to the mining one thing we treasure quite a lot: the Data Analytics
- 11. Analytics – is the process of discovery & communication of meaningful patterns in data. Reporting - extraction of the aggregated information for further analysis. Querying - data extraction process Analytics Faces
- 12. Traditional Operational/Analytics architecture Key issues Complex implementation Requires two servers (capital expenditures and operational expenditures) Data latency in analytics More businesses demand; requires real-time analytics IIS Server BI analysts
- 13. • Costs • Integration Problems (data types, constraints, network problems, etc) • Delay for getting the actual data Traditional Operational Analytics Problems:
- 14. Nothing substitutes analytics queries performance, possible using schemas customized (Star/Snowflake) and/or pre-aggregated cubes Modern Operational Analytics notes:
- 15. Minimizing data latency for analytics Benefits No data latency No ETL No separate data warehouse Challenges Analytics queries are resource intensive and can cause blocking Minimizing impact on operational workloads Sub-optimal execution of analytics on relational schema IIS Server BI analysts
- 16. Optimizing data latency for analytics BI analysts What is operational analytics and what does it mean to you? Operational analytics with disk-based tables Operational analytics with In-Memory OLTP IIS Server
- 17. There are now 2 types of Operational Analytics: Operational Analytics for Disk-Based Operational Analytics for InMemory Operational Analytics: SQL Server 2016
- 18. Updatable Columnstore SQL Server 2014
- 20. Operational Analytics Query Processing
- 22. Filtered Operational Analytics Create a filtered Nonclustered Columnstore Index!
- 23. Operational Analytics Demo with a Filtered Index
- 25. In-Memory Operational Analytics sys.sp_memory_optimized_cs_migration – compresses data from InMemory OLTP Table into In-Memory Columnstore
- 28. Compression Delay Warm Data Hot Data Recent DataArchive Data Compressed Row Groups Delta-Stores
- 29. The Problem: When we move a row from Delta-Store into the compressed Row Group, it becomes compressed. Once it is updated, the row is marked as deleted in the Deleted Bitmap/Deleted Buffer and the new value is inserted in Delta-Store.
- 30. The Solution is the Compression Delay Default delay is 0 min alter index PK_NCCi_test_inline on dbo.ncci_test_inline set (COMPRESSION_DELAY = 60 Minutes);
- 32. Clustered Columnstore (DWH) Nonclustered Indexes Primary & Foreign Keys + Unique Constraints Nonclustered Secondary Rowstore Locking
- 33. T-SQL Columnstore in SQL 2016
- 34. New T-SQL • Inline Table definition of Clustered Columnstore Index • Inline Table definition of Nonclustered Columnstore Index • In-Memory OLTP & Columnstore Indexes
- 35. New T-SQL for Disk-Based Columnstore CREATE TABLE [dbo].[FactOnlineSales_CCI]( [OnlineSalesKey] [int] NOT NULL, [StoreKey] [int] NOT NULL, [ProductKey] [int] NOT NULL, [PromotionKey] [int] NOT NULL, [CurrencyKey] [int] NOT NULL, [CustomerKey] [int] NOT NULL, INDEX PK_FactOnlineSales_CCI CLUSTERED COLUMNSTORE ); CREATE TABLE [dbo].[FactOnlineSales_NCCI]( [OnlineSalesKey] [int] NOT NULL, [StoreKey] [int] NOT NULL, [ProductKey] [int] NOT NULL, [PromotionKey] [int] NOT NULL, [CurrencyKey] [int] NOT NULL, [CustomerKey] [int] NOT NULL, INDEX PK_FactOnlineSales_CCI NONCLUSTERED COLUMNSTORE (OnlineSalesKey, CustomerKey) );
- 36. New T-SQL for In-Memory Columnstore CREATE TABLE [dbo].[FactOnlineSales_Hekaton]( [OnlineSalesKey] [int] NOT NULL, [StoreKey] [int] NOT NULL, [ProductKey] [int] NOT NULL, [PromotionKey] [int] NOT NULL, [CurrencyKey] [int] NOT NULL, [CustomerKey] [int] NOT NULL, Constraint PK_FactOnlineSales_Hekaton PRIMARY KEY NONCLUSTERED ([OnlineSalesKey]), INDEX CCI_FactOnlineSales_Hekaton CLUSTERED COLUMNSTORE ) WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA); ALTER TABLE dbo. FactOnlineSales_Hekaton ADD INDEX CCI_FactOnlineSales_Hekaton CLUSTERED COLUMNSTORE;
- 39. High Availability Readable Secondaries for Availability Groups through Snapshot & Read Committed Snapshot Isolation Levels support
- 40. Transactional Replication We can now control, how the HTAP Columnstore Indexes are being replicated – if they should be created on the Subscriber or not.
- 42. Further Columnstore Improvements • Data Loading Improvements • Batch Mode • Performance Improvements • Change Tracking • Maintenance Improvements • Monitoring Improvements (DMV, Extended Events, Performance Counters)
- 43. Data Loading Improvements • Parallel Data Loading finally enabled • SIMD support • Delta-Stores are not Page-Compressed!!! • Under NCCI Delta-Stores are automatically increasing their max size with each iteration under pressure • Compression Delay • SSIS • Minimal Logging
- 45. SIMD Improvements Columnstore is SIMD-aware and uses it for different purposes, such as: - Huffman decoding - Predicate Pushdown - String Hash
- 46. Delta-Stores for NCCI - Under memory pressure the size of the created Delta-Stores for the Nonclustered Columnstore Indexes will be doubled for each succeeding operation from 1048576 rows to ~2 Million Rows, ~4 Million Rows, ... 32 Million Rows
- 47. The solution: Compression Delay Default delay is 0 min alter index PK_NCCi_test_inline on dbo.ncci_test_inline set (COMPRESSION_DELAY = 60 Minutes);
- 48. SSIS 2016 Improvements AutoAdjustBufferSize - True Warning: It’s not enough, you will need to set up the maximum number of rows per buffer, otherwise your performance will be extremely slow! DefaultBufferMaxRows = 1048576
- 49. Minimal Logging
- 50. Minimal Logging
- 52. Dynamic Memory Grants Trace Flag 9389 - Enables dynamic memory grant for batch mode operators. Memory Grant Memory Required Dynamic Memory Grant
- 54. Batch Mode Improvements • Batch Mode support for 1 core execution plan operators • Batch Mode support for the Sort operator • Batch Mode support for the Multiple Distinct Count operations • Batch Mode support for the Left Anti-Semi Join operators
- 55. Batch Sorting Improvements Trace Flag Description 9347 Disables batch mode sort operator 9349 Disables batch mode top sort operator 9358 Disable batch mode sort operations in a complex parallel query in SQL Server 2016
- 56. Batch Mode Demo
- 57. Batch Mode for Windowing Functions
- 59. Further Batch Improvements • Simple Aggregate Predicate Pushdown • Local Aggregation • String Predicate Pushdown for Index Scan operator in Batch Mode
- 63. Change Tracking / CDC in SQL 2016 Columnstore Change Tracking Change Data Capture Temporal (2016+) Clustered No No Yes Nonclustered Yes Yes Yes Memory-Optimized No No Yes
- 64. Maintenance Improvements • Better ALTER INDEX … REORGANIZE (removes deleted rows, less memory pressure) • Self-Merging (this is when Row Group is being recompressed eliminating obsolete row versions that are found in Deleted Bitmap) • Intergroup Merging (when the sum of active rows between multiple Row Groups is less then 1048576, then their information can be merged into a new single Row Group)
- 65. Maintenance Improvements • Being a compressed Row Group (Delta-Stores & Tail Row Groups are not qualified) • A Row Group has to have at least 10% of its data being marked as obsolete in Deleted Bitmap (for In-Memory this threshold is set on 90%) • – A Row Group should not being trimmed because of the dictionary pressure.
- 66. Row Group Merging in SQL Server 2016 Columnstore Self-Merge Intergroup Merge Cleanup Clustered Yes Yes Yes Nonclustered Yes Yes*1 Yes Memory-Optimized Yes*2 No Yes
- 67. New DMVs: • sys.dm_column_store_object_pool • sys.dm_db_column_store_row_group_physical_stats • sys.dm_db_column_store_row_group_operational_stats • sys.internal_partitions
- 68. Enhanced DMVs: • sys.dm_db_index_operational_stats • sys.dm_db_index_physical_stats
- 69. Extended Events
- 70. Extended Events Categories 2016 • Columnstore Index creation • Batch Mode • Tuple Mover • Columnstore Internal Object Operations • Columnstore Processing • Columnstore Object Pool • Errors
- 71. Configuration Improvement Large Pages (Trace Flag 834) should be supported in SQL Server 2016, or at least the Connect Item for it is closed as fixed. Take them to DEV & QA Environments and let Microsoft know the results
- 72. New Trace Flags Trace Flag Description 9347 Disables batch mode sort operator 9349 Disables batch mode top sort operator 9358 Disable batch mode sort operations in a complex parallel query in SQL Server 2016 9389 Enables dynamic memory grant for batch mode operators 10204 Disables merge/recompress during columnstore index reorganization
- 73. Indexed Views SQL Server 2016 supports Indexed Views with updatable Nonclustered Columnstore Indexes!
- 74. Resources: My Columnstore Blogpost Series: http://www.nikoport.com/columnstore CISL – Open Source Columnstore Library: https://github.com/NikoNeugebauer/CISL
- 75. Session Evaluations ways to access Go to passSummit.com Download the GuideBook App and search: PASS Summit 2016 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide Submit by 5pm Friday November 6th to WIN prizes Your feedback is important and valuable. 3
- 76. Thank You Learn more from Niko Neugebauer [email protected] or follow @NikoNeugebauer