Partitioning your Oracle Data Warehouse - Just a simple task?

Partitioning Your Oracle Data Warehouse
– Just a Simple Task?
Dani Schnider
Principal Consultant
Business Intelligence
dani.schnider@trivadis.com
Oracle Open World 2009,
San Francisco
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
2014 © Trivadis
2009
1

Dani Schnider
 Principal Consultant und
DWH/BI Lead Architect
bei Trivadis in Zürich
 Kursleiter für Trivadis-Kurse
über Data Warehousing,
SQL Optimierung und Oracle
Warehouse Builder
 Co-Autor des Buches «Data
Warehousing mit Oracle»
2014 © Trivadis
2
2009

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort
2014 © Trivadis
11 Trivadis Niederlassungen mit
über 600 Mitarbeitenden
200 Service Level Agreements
Mehr als 4'000 Trainingsteilnehmer
Forschungs- und Entwicklungs-budget:
CHF 5.0 / EUR 4 Mio.
Finanziell unabhängig und
nachhaltig profitabel
Erfahrung aus mehr als 1'900
Projekten pro Jahr bei über 800
Kunden
Stand 12/2012
Hamburg
Düsseldorf
Frankfurt
Freiburg
München
Wien
Basel
Bern Zürich
Lausanne
3
Stuttgart
2009
3

AGENDA
1. Partitioning Concepts
2. The Right Partition Key
3. Large Dimensions
4. Partition Maintenance
2014 © Trivadis
2009
4

Partitioning – The Basic Idea
 Decompose tables/indexes into smaller pieces
2014 © Trivadis
2009
5

Partitioning Methods in Oracle
 Additionally: Interval Partitioning, Reference Partitioning, Virtual
Column-Based Partitioning
2014 © Trivadis
2009
6

Benefits of Partitioning
Partition Pruning
 Reduce I/O: Only relevant partitions have to be accessed
Partition-Wise Joins
 Full PWJ: Join two equi-partitioned tables (parallel or serial)
 Partial PWJ: Join partitioned with non-partitioned table (parallel)
Rolling History
 Create new partitions in periodical intervals
 Drop old partitions in periodical intervals
Manageability
 Backups, statistics gathering, compressing on individual partitions
2014 © Trivadis
2009
7

AGENDA
3. Large Dimensions
2014 © Trivadis
2009
8

Partition Methods and Partition Keys
 Important questions for Partitioning:
 Which tables should be partitioned?
 Which partition method should be used?
 Which partition key(s) should be used?
 Partition key is important for
 Query optimization (partition pruning, partition-wise
2014 © Trivadis
joins)
 ETL performance (partition exchange, rolling
history)
 Typically in data warehouses
 RANGE partitioning of fact tables on DATE
column
 But which DATE column?
2009
9

Practical Example 1: Airline Company
 Flight bookings are stored in partitioned table
 RANGE partitioning per month, partition key: booking date
 Problem: Most of the reports are based on the flight date
 Flights can be booked 11 months ahead
 11 partitions must be read of one particular flight date
2014 © Trivadis
2009
10

 Solution: partition key flight date instead of booking date
 Data is loaded into current and future partitions
 Reports based on flight date read only one partition
 Reports based on booking date must read 11 (small) partitions
2014 © Trivadis
2009
11

 Better solution: Composite RANGE-RANGE partitioning
 RANGE partitioning on flight date
 RANGE subpartitioning on booking date
 More flexibility for reports on flight date and/or on booking date
2014 © Trivadis
2009
12

Practical Example 2: International Bank
 Account balance data of international customers
 Monthly files of different locations (countries)
 Correction files replace last delivery for the same month/country
 Original approach
 Technical load id for each combination of month/country
 LIST partitioning on load id
 Files are loaded in stage table
 Partition exchange with current partition
2014 © Trivadis
2009
13

Practical Example 2: International Bank
 Problem: partition key load id is useless for queries
 Queries are based on balance date
2014 © Trivadis
 Solution
 RANGE partitioning on balance date
 LIST subpartitions on country code
 Partition exchange with subpartitions
2009
14

AGENDA
3. Large Dimensions
2014 © Trivadis
2009
15

Partitioning in Star Schema
2014 © Trivadis
 Fact Table
 Usually „big“ (millions to billions of rows)
 RANGE partitioning by DATE column
 Dimension Tables
 Usually „small“ (10 to 10000 rows)
 In most cases not partitioned
 But how about large dimensions?
 e.g. customer dimension with millions of
rows
2009
16

HASH Partitioning on Large Dimension
 DIM_CUSTOMER
2014 © Trivadis
2009
17
 HASH Partitioning
 Partition Key: CUSTOMER_ID
 FCT_SALES
 Composite RANGE-HASH Partitioning
 Partition Key: SALES_DATE
 Subpartition Key: CUSTOMER_ID

HASH Partitioning on Large Dimension
2014 © Trivadis
2009
18

Practical Example 3: Telecommunication Company
2014 © Trivadis
2009
19

LIST Partitioning on Large Dimension
 DIM_CUSTOMER
2014 © Trivadis
2009
20
 LIST Partitioning
 Partition Key: COUNTRY_CODE
 FCT_SALES
 Composite RANGE-LIST Partitioning
 Partition Key: SALES_DATE
 Subpartition Key: COUNTRY_CODE
(denormalized column in fact table)

LIST Partitioning on Large Dimension
2014 © Trivadis
2009
21

 New approach for partition pruning on join conditions
 A bloom filter is created based on the dimension table restriction
 Dynamic partition pruning based on that bloom filter
2014 © Trivadis
2009
22

AGENDA
3. Large Dimensions
2014 © Trivadis
2009
23

Practical Example 3: Monthly Partition Maintenance
 Requirements
 Monthly partitions on all fact tables, daily inserts into current partitions
 3 years of history (36 partitions per table)
 Table compression to increase full table scan performance
 Backup of current partitions only
2014 © Trivadis
2009
24

Gathering Optimizer Statistics
 DBMS_STATS parameter GRANULARITY defines statistics level
2014 © Trivadis
 Example:
2009
31

Problem of Global Statistics
 Global statistics are essential for good execution plans
 num_distinct, low_value, high_value, density, histograms
 Gathering global statistics is time-consuming
 All partitions must be scanned
 Typical approach
 After loading data, only modified partition statistics are gathered
 Global statistics are gathered on regular time base (e.g. weekends)
2014 © Trivadis
2009
32

Incremental Global Statistics
 Synopsis-based gathering of statistics
 For each partition a synopsis is stored in SYSAUX tablespace
 Statistics metadata for partition and columns of partition
 Global statistics by aggregating the synopses from each partition
 Activate incremental global statistics:
2014 © Trivadis
2009
33

Partitioning Your Data Warehouse – Core Messages
2014 © Trivadis
 Oracle Partitioning is a powerful option
– not only for data warehouses
 The concept is simple, but the reality
can be complex
 Many new partitioning features added
in Oracle Database 11g
 New Composite Partitioning methods
 Interval Partitioning
 Join-Filter Pruning
 Incremental Global Statistics
2009
34

Thank you.
Dani Schnider
Principal Consultant
Business Intelligence
dani.schnider@trivadis.com
Oracle Open World 2009,
San Francisco
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
2014 © Trivadis
2009
35

Partitioning your Oracle Data Warehouse - Just a simple task?

More Related Content

What's hot (19)

Similar to Partitioning your Oracle Data Warehouse - Just a simple task? (20)

More from Trivadis (20)

Recently uploaded (16)

Partitioning your Oracle Data Warehouse - Just a simple task?