Flexible Indexing with Postgres

Flexible Indexing with Postgres
BRUCE MOMJIAN
December, 2014
Postgres offers a wide variety of indexing structures, and many
index lookup methods with specialized capabilities.This talk
explores the many Postgres indexing options. Includes concepts from
Teodor Sigaev, Alexander Korotkov, Oleg Bartunov, Jonathan Katz
Creative Commons Attribution License http://momjian.us/presentations
1 / 53

PostgreSQL the database…
◮ Open Source Object Relational DBMS since 1996
◮ Distributed under the PostgreSQL License
◮ Similar technical heritage as Oracle, SQL Server & DB2
◮ However, a strong adherence to standards (ANSI-SQL 2008)
◮ Highly extensible and adaptable design
◮ Languages, indexing, data types, etc.
◮ E.g. PostGIS, JSONB, SQL/MED
◮ Extensive use throughout the world for applications and
organizations of all types
◮ Bundled into Red Hat Enterprise Linux, Ubuntu, CentOS
and Amazon Linux
Flexible Indexing with Postgres 2 / 53

PostgreSQL the community…
◮ Independent community led by a Core Team of six
◮ Large, active and vibrant community
◮ www.postgresql.org
◮ Downloads, Mailing lists, Documentation
◮ Sponsors sampler:
◮ Google, Red Hat, VMWare, Skype, Salesforce, HP and
EnterpriseDB
◮ http://www.postgresql.org/community/

EnterpriseDB the company…
◮ The worldwide leader of Postgres based products and
services founded in 2004
◮ Customers include 50 of the Fortune 500 and 98 of the
Forbes Global 2000
◮ Enterprise offerings:
◮ PostgreSQL Support, Services and Training
◮ Postgres Plus Advanced Server with Oracle Compatibility
◮ Tools for Monitoring, Replication, HA, Backup & Recovery
Community
◮ Citizenship
◮ Contributor of key features: Materialized Views, JSON, &
more
◮ Nine community members on staff

EnterpriseDB the company…

Outline
1. Traditional indexing
2. Expression indexes
3. Partial indexes
4. Beneﬁts of bitmap index scans
5. Non-b-tree index types
6. Data type support for index types
7. Index usage summary

Traditional Indexing
https://www.ﬂickr.com/photos/ogimogi/

B-Tree
◮ Ideal for looking up unique values and maintaining unique
indexes
◮ High concurrency implementation
◮ Index is key/row-pointer, key/row-pointer
◮ Supply ordered data for queries
◮ ORDER BY clauses (and LIMIT)
◮ Merge joins
◮ Nested loop with index scans

But I Want More!
◮ Index expressions/functions
◮ Row control
◮ Index non-linear data
◮ Closest-match searches
◮ Index data with many duplicates
◮ Index multi-valued ﬁelds

Expression Index
SELECT * FROM customer WHERE lower(name) = ’andy’;
CREATE INDEX i_customer_name ON customer (name);
CREATE INDEX i_customer_lower ON customer (lower(name));

Let’s Test It
CREATE TABLE customer (name) AS
SELECT ’cust’ || i
FROM generate_series(1, 1000) AS g(i);
SELECT 1000
CREATE INDEX i_customer_name ON customer (name);
CREATE INDEX
EXPLAIN SELECT * FROM customer WHERE name = ’cust999’;
QUERY PLAN
------------------------------------------------------
Index Only Scan using i_customer_name on customer ...
Index Cond: (name = ’cust999’::text)
EXPLAIN SELECT * FROM customer WHERE lower(name) = ’cust999’;
QUERY PLAN
---------------------------------------------------------
Seq Scan on customer (cost=0.00..20.00 rows=5 width=7)
Filter: (lower(name) = ’cust999’::text)

Create an Expression Index
CREATE INDEX i_customer_lower ON customer (lower(name));
CREATE INDEX
EXPLAIN SELECT * FROM customer WHERE lower(name) = ’cust999’;
QUERY PLAN
---------------------------------------------------------------
Bitmap Heap Scan on customer (cost=4.32..9.66 rows=5 width=7)
Recheck Cond: (lower(name) = ’cust999’::text)
-> Bitmap Index Scan on i_customer_lower ...
Index Cond: (lower(name) = ’cust999’::text)

Other Expression Index Options
◮ User-deﬁned functions
◮ Concatenation of columns
◮ Math expressions
◮ Only IMMUTABLE functions can be used
◮ Consider casting when matching WHERE clause expressions
to the indexed expression

Partial Index: Index Row Control
◮ Why index every row if you are only going to look up some
of them?
◮ Smaller index on disk and in memory
◮ More shallow index
◮ Less INSERT/UPDATE index overhead
◮ Sequential scan still possible

Partial Index Creation
ALTER TABLE customer ADD COLUMN state CHAR(2);
ALTER TABLE
UPDATE customer SET state = ’AZ’
WHERE name LIKE ’cust9__’;
UPDATE 100
CREATE INDEX i_customer_state_az ON customer (state) WHERE state = ’AZ’;
CREATE INDEX

Test the Partial Index
EXPLAIN SELECT * FROM customer WHERE state = ’PA’;
QUERY PLAN
----------------------------------------------------------
Filter: (state = ’PA’::bpchar)
EXPLAIN SELECT * FROM customer WHERE state = ’AZ’;
QUERY PLAN
----------------------------------------------------------------------------
Recheck Cond: (state = ’AZ’::bpchar)
-> Bitmap Index Scan on i_customer_state_az ...
Index Cond: (state = ’AZ’::bpchar)

Partial Index With Different Indexed Column
DROP INDEX i_customer_name;
DROP INDEX
CREATE INDEX i_customer_name_az ON customer (name) WHERE state = ’AZ’;
CREATE INDEX
EXPLAIN SELECT * FROM customer WHERE name = ’cust975’;
QUERY PLAN
----------------------------------------------------------
Filter: (name = ’cust975’::text)
Index Cond: (state = ’AZ’::bpchar)

Partial Index With Different Indexed Column
EXPLAIN SELECT * FROM customer
WHERE name = ’cust975’ AND state = ’AZ’;
QUERY PLAN
-----------------------------------------------------
Index Scan using i_customer_name_az on customer ...
Index Cond: (name = ’cust975’::text)
EXPLAIN SELECT * FROM customer
WHERE state = ’AZ’;
QUERY PLAN
----------------------------------------------------------------
Recheck Cond: (state = ’AZ’::bpchar)
-> Bitmap Index Scan on i_customer_name_az ...

Beneﬁts of Bitmap Index Scans
◮ Used when:
◮ an index lookup might generate multiple hits on the same
heap (data) page
◮ using multiple indexes for a single query is useful
◮ Creates a bitmap of matching entries in memory
◮ Row or block-level granularity
◮ Bitmap allows heap pages to be visited only once for
multiple matches
◮ Bitmap can merge the results from several indexes with
AND/OR ﬁltering
◮ Automatically enabled by the optimizer

Bitmap Index Scan
=&
Combined
’A’ AND ’NS’
1
0
1
0
TableIndex 1
col1 = ’A’
Index 2
1
0
0
col2 = ’NS’
1 0
1
0
0
Index

Non-B-Tree Index Types
https://www.ﬂickr.com/photos/archeon/

Generalized Inverted Index (GIN)
◮ Best for indexing values that can contain multiple keys, e.g.
◮ text documents
◮ JSON
◮ multi-dimensional data
◮ Also ideal for columns that contain many duplicates
◮ Key is stored only once
◮ Index is key/many-row-pointers
◮ Index updates are batched, though always checked for
accuracy
◮ In Postgres 9.4
◮ compression of row pointer list
◮ optimized multi-key ﬁltering

Generalized Search Tree (GIST)
GIST is a general indexing framework designed to allow indexing
of complex data types with minimal programming. Supported
data types include:
◮ geometric types
◮ range types
◮ hstore (key/value pairs)
◮ intarray (integer arrays)
◮ pg_trgm (trigrams)
Supports optional “distance” for nearest-neighbors/closest
matches. (GIN is also generalized.)

Space-Partitioned Generalized Search Tree
(SP-GIST)
◮ Similar to GIST in that it is a generalized indexing framework
◮ Allows the key to be split apart (decomposed)
◮ Parts are indexed hierarchically into partitions
◮ Partitions are of different sizes
◮ Each child needs to store only the child-unique portion of
the original value because each entry in the partition shares
the same parent value.

Hash Indexes
◮ Equality, non-equality lookups; no range lookups
◮ Not crash-safe
◮ Not replicated
◮ Cannot be restored via point-in-time recovery
◮ Poor performance and concurrency characteristics
◮ Boo

I Am Not Making This Up
SELECT amname, obj_description(oid, ’pg_am’)
FROM pg_am ORDER BY 1;
amname | obj_description
--------+-----------------------------
btree | b-tree index access method
gin | GIN index access method
gist | GiST index access method
hash | hash index access method
spgist | SP-GiST index access method

Data Type Support for Index Types
https://www.ﬂickr.com/photos/jonobass/

Finding Supported Data Types - B-Tree
SELECT opfname FROM pg_opfamily, pg_am
WHERE opfmethod = pg_am.oid AND amname = ’btree’ ORDER BY 1;
abstime_ops jsonb_ops text_ops
array_ops macaddr_ops text_pattern_ops
bit_ops money_ops tid_ops
bool_ops name_ops time_ops
bpchar_ops network_ops timetz_ops
bpchar_pattern_ops numeric_ops tinterval_ops
bytea_ops oid_ops tsquery_ops
char_ops oidvector_ops tsvector_ops
datetime_ops pg_lsn_ops uuid_ops
enum_ops range_ops varbit_ops
float_ops record_image_ops
integer_ops record_ops
interval_ops reltime_ops
These data types are mostly single-value and easily ordered.
B-tree support for multi-valued types like tsvector is only for
complete-ﬁeld equality comparisons.

Finding Supported Data Types - GIN
WHERE opfmethod = pg_am.oid AND amname = ’gin’ ORDER BY 1;
opfname
----------------
array_ops
jsonb_ops
jsonb_path_ops
tsvector_ops
These date types are multi-value, where each value is
independent.

Finding Supported Data Types - GIST
WHERE opfmethod = pg_am.oid AND amname = ’gist’ ORDER BY 1;
opfname
--------------
box_ops
circle_ops
jsonb_ops
network_ops
point_ops
poly_ops
range_ops
tsquery_ops
tsvector_ops
These date types are multi-value — some have independent
values (JSON, tsvector), others have dependent values (point,
box).

Finding Supported Data Types - SP-GIST
WHERE opfmethod = pg_am.oid AND amname = ’spgist’ ORDER BY 1;
opfname
----------------
kd_point_ops
quad_point_ops
range_ops
text_ops
For text, this is useful when the keys are long.

Index Type Examples
https://www.ﬂickr.com/photos/samcatchesides/

B-Tree
C
Item Item Item
Special< N< F
>= N
G
Internal
Leaf
Page Header Item Item Item
SpecialC
E
A
Heap
Page Header Item Item Item
SpecialK
L
G
M I A E P K W L
Page Header

GIN Example Using tsvector_ops
CREATE TABLE articles (doc TSVECTOR);
CREATE TABLE
INSERT INTO articles VALUES (’The fox is sick’);
INSERT 0 1
INSERT INTO articles VALUES (’How sick is this’);
INSERT 0 1
SELECT ctid, * FROM articles ORDER BY 1;
ctid | doc
-------+---------------------------
(0,1) | ’The’ ’fox’ ’is’ ’sick’
(0,2) | ’How’ ’is’ ’sick’ ’this’

GIN Example Using tsvector_ops
SELECT ctid, * FROM articles ORDER BY 1;
ctid | doc
-------+---------------------------
(0,1) | ’The’ ’fox’ ’is’ ’sick’
(0,2) | ’How’ ’is’ ’sick’ ’this’
fox (0,1)
is (0,1), (0,2)
sick (0,1), (0,2)
this (0,2)
How (0,2)
The (0,1)
Integer arrays are indexed similarly.

GIN Example Using JSON
CREATE TABLE webapp (doc JSON);
CREATE TABLE
INSERT INTO webapp VALUES
(’{"name" : "Bill", "active" : true}’);
INSERT 0 1
INSERT INTO webapp VALUES
(’{"name" : "Jack", "active" : true}’);
INSERT 0 1
SELECT ctid, * FROM webapp ORDER BY 1;
ctid | doc
-------+------------------------------------
(0,1) | {"name" : "Bill", "active" : true}
(0,2) | {"name" : "Jack", "active" : true}

GIN Example Using jsonb_ops (default)
CREATE INDEX i_webapp_yc ON webapp
USING gin (doc /* jsonb_ops */);
active (0,1), (0,2)
name (0,1), (0,2)
true (0,1), (0,2)
Bill (0,1)
Jack (0,2)

GIN Example Using jsonb_path_ops
CREATE INDEX i_webapp_doc_path ON webapp
USING gin (doc jsonb_path_ops);
hash(active, true) (0,1), (0,2)
hash(name, Bill) (0,1)
hash(name, Jack) (0,2)
Nested keys have their parent keys (paths) prepended before
hashing.

GIST
◮ Supports data types with loosely-coupled values, like
tsvector, JSONB
◮ Uniquely supports data types with tightly-coupled values
◮ multi-dimensional types (geographic)
◮ range types
◮ IP network data type

Linear Indexing
0 5−5
0 5−5
0 5−5
= 2
>= 2

Multi-Dimensional
5−5
x
y

Linear Methods Are Inefﬁcient
5−5
x
y
x >= 2
x

R-Tree Indexes Bounding Boxes
5−5
x
y
x
Level 1
Level 3
Level 2
Geographic objects (lines, polygons) also can appear in r-tree
indexes. based on their own bounding boxes.

GIST Two-Dimensional Ops
box_ops
circle_ops
point_ops
poly_ops
PostGIS also uses this indexing method.

Range Indexing With GIST
GIST range type indexing uses large ranges at the top level of the
index, with ranges decreasing in size at lower levels, just like how
r-tree bounding boxes are indexed.

SP-GIST TEXT_OPS Example (Sufﬁx Tree)
yahoo.com/google.com/
http://
index.html
maps.html
index.html
flickr.html cgi.html
google.com/public/ berkeley.edu/
README
ftp://
bin.tar.gz
doc.pdf
Internally split by character. B-trees use range partitioning, e.g.
A-C, rather than common preﬁx partitioning, so a btree key must
store the full key value.

Other SP-GIST Index Examples
◮ quad_point_ops uses four corner points in square partitions of
decreasing size
◮ kd_point_ops splits on only one dimension

Extension Index Support
◮ btree_gin (GIN)
◮ btree_gist (GIST)
◮ cube (GIST)
◮ hstore (GIST, GIN)
◮ intarray (GIST, GIN)
◮ ltree (GIST)
◮ pg_trgm (GIST, GIN)
◮ PostGIS
◮ seg

Index Usage Summary
https://www.ﬂickr.com/photos/jubilo/

When To Create Indexes
◮ pg_stat_user_tables.seq_scan is high
◮ Check frequently-executed queries with EXPLAIN (ﬁnd via
pg_stat_statements or pgbadger)
◮ Squential scans are not always bad
◮ If pg_stat_user_indexes.idx_scan is low,the index might be
unnecessary
◮ Unnecessary indexes use storage space and slow down
INSERTs and some UPDATEs

Evaluating Index Types
◮ Build time, INSERT/UPDATE overhead
◮ Storage size
◮ Access speed
◮ Operator lookup ﬂexibility

Additional Resources…
◮ Postgres Downloads:
◮ www.enterprisedb.com/downloads
◮ Product and Services information:
◮ info@enterprisedb.com

Conclusion
http://momjian.us/presentations https://www.ﬂickr.com/photos/philipp_zurmoehle/

Flexible Indexing with Postgres

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Flexible Indexing with Postgres (20)

More from EDB (20)

Recently uploaded (20)

Flexible Indexing with Postgres