Partition and conquer large data in PostgreSQL 10
4 likes2,108 views
This document discusses declarative partitioning in PostgreSQL 10, which allows tables to be partitioned for better manageability, performance, and storage optimization. Key points include: - Tables can be partitioned by range or list on single or multiple columns, with sub-partitioning support. - Partitions act as independent tables that can have their own constraints, indexes, and tablespaces. Utilities can be run individually on partitions. - Queries benefit from partition pruning, partition-wise joins and aggregations for better performance. DMLs are faster with less overhead.
Partition and conquer large data in PostgreSQL 10
- 1. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 1 Partition and Conquer large data in PostgreSQL 10 • Ashutosh Bapat | 2017.02.03 @ PGConf India
- 2. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 2 • Manageability – Smaller data is easier to manage – Partition-wise utility commands – Easy bulk load and deletes • Query/DML Performance – Partition elimination (pruning) – Partition-wise joins, aggregate – Distribute DMLs across partitions • Data storages based on partition properties – “hot” and “cold” partitions – Foreign partitions Why to partition tables?
- 3. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 3 • Maintenance – No triggers, rules or constraints to maintain – Adding/deleting partition does not require more than a single command • Faster queries and DMLs – No trigger execution overhead ● DMLs are 10-20 times faster – Simpler partition bounds representation – Faster partition pruning – Partition-wise join, aggregation Why declarative partitioning? (instead of inheritance based partitioning)
- 4. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 4 Declarative partitioning in PostgreSQL 10
- 5. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 5 ● Partitioning methods: List, range, (WIP. hash) ● Partition key: Single or multiple columns, expressions ● Sub-partitioning – Partitioned partitions – Mixed sub-partitioning ● Development efforts – Several previous attempts by many people – Amit Langote proposed first patch in August 2015 – Got committed in Dec 2016, bug fixes, doc changes continue ... PostgreSQL Declarative partitioning
- 6. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 6 CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1); Creating partitioned table part_tab (c1 int, c2 int, ...)
- 7. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 7 CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1); CREATE TABLE part1 PARTITION OF part_tab FOR VALUES FROM (0) TO (100); CREATE TABLE part2 PARTITION OF part_tab FOR VALUES FROM (100) TO (200); Creating partitioned table part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 Tablespaces
- 8. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 8 ALTER TABLE part_tab ATTACH PARTITION ext_part FOR VALUES FROM (400) to (500); ATTACH partition part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 ext_part 400 to 500 ext_part (c1 int, c2 int, ...) CHECK c1 >= 400 AND c1 < 500 Bulk load data.csv
- 9. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 9 ALTER TABLE part_tab DETACH PARTITION ext_part; DETACH partition part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 ext_part (c1 int, c2 int, ...) CHECK c1 >= 400 AND c1 < 500
- 10. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 10 ● Same columns as parent table ● Partition specific constraints, defaults, indexes, storage parameters ● Tablespace separate from that of parent – “hot” and “cold” partitions ● VACUUM, ANALYZE, CLUSTER can be run separately – Utilities don't block the whole table – Work where it's required Partitions are tables
- 11. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 11 CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1); ... CREATE FOREIGN TABLE part3 PARTITION OF part_tab FOR VALUES FROM (300) TO (400) SERVER xyz_server; CREATE FOREIGN TABLE part4 PARTITION OF part_tab FOR VALUES FROM (400) TO (500) SERVER abc_server; Foreign partitions part_tab (c1 int, c2 int, ...) Part1 0 to 100 Part2 100 to 200 Part3 300 to 400 fpart3 (c1 int, c2 int, ...) c1 >= 300 AND c1 < 400 xyz_server xyz_fdw Part4 400 to 500 fpart3 (c1 int, c2 int, ...) c1 >= 400 AND c1 < 500 abc_server abc_fdw
- 12. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 12 Query optimizations partition pruning partition-wise join partition-wise aggregation
- 13. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 13 Query optimization: partition pruning SELECT * FROM t1 WHERE c1 = 350; Partition 1 FOR VALUES FROM (0) TO (100) Partitioned table t1 (c1 int, c2 int, …) Partition 4 FOR VALUES FROM (300) TO (400) Partition 3 FOR VALUES FROM (200) TO (300) Partition 2 FOR VALUES FROM (100) TO (200) SELECT * FROM t1 WHERE c1 BETWEEN 150 AND 250; Constraintexclusion
- 14. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 14 Query optimization: partition-wise join Partition 1 FOR VALUES (0) TO (100) Partitioned table t1 (c1 int, ...) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) Partition 1 FOR VALUES (0) TO (100) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) Partition 1 FOR VALUES (0) TO (100) Partitioned table t2 (c1 int, ...) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) t1 JOIN t2 ON t1.c1 = t2.c1 Partitioned join Partition 3 FOR VALUES (200) TO (300) Partition 1 FOR VALUES (0) TO (100) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200) Partition 3 FOR VALUES (200) TO (300) Partition 1 FOR VALUES (0) TO (100) Partition 3 FOR VALUES (200) TO (300) Partition 2 FOR VALUES (100) TO (200)
- 15. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 15 Query optimization: partition-wise aggregation Partitioned table t1 (c1 int, ...) Partition 1 Partition 3 Partition 2 Partition 1 Partitioned table t2 (c1 int, ...) Partition 3 Partition 2 t1 JOIN t2 ON t1.c1 = t2.c1 Partition 1 Partition 3 Partition 2 Partition 1 Partition 3 Partition 2 Agg/Group Agg/Group Agg/Group Agg/Group
- 16. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 16 • Bad partitioning is worse than no partitioning • Query optimization – Partition key is the key to success – Columns in conditions – include it in the query • Storage management – Columns that decide the storage – Usually timestamp of the row – Easy to add and drop partitions • Keep an eye on current limitations – Expected to reduce with next few releases Choosing partitioning scheme
- 17. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 17 • We have just begun … • No triggers on partitioned table – Need to create those for each of the partitions • No SPLIT, MERGE, EXCHANGE partition • No global indexes – No primary key, unique constraints • Foreign partitions – INSERTs via parent table not supported – No ACID support for transactions involving multiple foreign servers Limitations