(Ab)using 4d Indexing

(Ab)using 4D indexing in
PostGIS 2.2 with
PostgreSQL 9.5 to give you
the perfect match
Victor Blomqvist
vb@viblo.se
blomqvist@tantanapp.com
Tantan (探探)
March 19, pgDay Asia 2016 in Singapore
1tantanapp.com

At Tantan we use PostgreSQL &
PostGIS for everything!
2tantanapp.com

Suggesting users is difficult
• How should we rank them?
• How can it execute quickly?
(At Tantan we need to do 1000 ranking queries per
second at peak!)
tantanapp.com 3

The exciting new feature in
PostGIS is this:
<<->>
tantanapp.com 4

Today I will show how to take
advantage of the 4th dimension!
1. Use double the amount of dimensions from 2
2. …
3. Profit!
tantanapp.com 5

We will look at 3 different properties
to help our suggestion SELECT
1. Popularity
2. Age
3. Activity
tantanapp.com 6

Let’s begin with 2 dimensions
tantanapp.com 7
Y
(Latitude)
X
(Longitude)

Table and index
CREATE TABLE users (
id serial PRIMARY KEY,
birthdate date,
location geometry,
active_time timestamp,
popularity double precision
);
CREATE INDEX users_location_gix
ON users USING GIST (location);
tantanapp.com 8

Selecting
SELECT * FROM users
ORDER BY location <-> ST_MakePoint(103.8, 1.3)
/* Singapore */
LIMIT 10;
tantanapp.com 9

Lets look at our first case
1. Popularity <--
2. Age
3. Activity
tantanapp.com 10

The popularity formula
Popularity = likes / (likes + dislikes)
tantanapp.com 11

Order by location and popularity
WITH x AS (SELECT * FROM users
LIMIT 100)
SELECT * FROM x
ORDER BY
ST_Distance(location::geography, ST_MakePoint(103.8, 1.3))
* 1 / (popularity+1)
LIMIT 10;
tantanapp.com 12

Picture of x-y-z axis
tantanapp.com 13
X
(Longitude)
Y
(Latitude)
Z
(Popularity)

Adding a 3rd dimension!
ALTER TABLE users ADD COLUMN loc_pop geometry;
UPDATE users
SET loc_pop = ST_makepoint(ST_X(location),
ST_Y(location), 0.01 * 1 / (popularity+1));
CREATE INDEX users_loc_pop_gix
ON users USING GIST (loc_pop gist_geometry_ops_nd);
tantanapp.com 14

To use our new 3D index we need
to use the new <<->> operator
<<->> — Returns the n-D distance between the centroids
of A and B bounding boxes.
This operand will make use of n-D GiST indexes that may
be available on the geometries. It is different from other
operators that use spatial indexes in that the spatial
index is only used when the operator is in the ORDER BY
clause.
* http://postgis.net/docs/manual-2.2/geometry_distance_centroid_nd.html
tantanapp.com 15

With <<->> our query becomes
SELECT * FROM users
ORDER BY loc_pop <<->> ST_MakePoint(103.8, 1.3, 0)
LIMIT 10;
tantanapp.com 16

Second case
1. Popularity
2. Age <--
3. Activity
tantanapp.com 17

A quick review of our table:
CREATE TABLE users (
id serial PRIMARY KEY,
birthdate date,
location geometry,
active_time timestamp,
popularity double precision
);
tantanapp.com 18

Selecting with age filter
SELECT * FROM users
WHERE age(birthdate) between '20 years' AND '30 years'
LIMIT 10;
tantanapp.com 19

A very selective user might only
want to look at 74 year olds
SELECT * FROM users
WHERE age(birthdate) between '74 years' AND '75 years'
LIMIT 10;
tantanapp.com 20

Explain analyze of the query
Limit (cost=0.41..62011.93 rows=10 width=96) (actual
time=49.456..551.955 rows=10 loops=1)
-> Index Scan using users_location_gix on users
(cost=0.41..1103805.51 rows=178 width=96) (actual time=49.398..546.631
rows=10 loops=1)
Order By: (location <-> 'XXX'::geometry)
Filter: ((birthdate >= (now() - '75 years'::interval)) AND (birthdate <=
(now() - '74 years'::interval)))
Rows Removed by Filter: 192609
Planning time: 0.157 ms
Execution time: 553.151 ms
tantanapp.com 21

This is a real
problem
tantanapp.com 22

Possible solutions?
• Prevent searches of
restricted ages
• Add a distance
restriction
• Add age to the geo index
tantanapp.com 23

Adding age to the geo index
ALTER TABLE users ADD COLUMN loc_age geometry;
UPDATE users
SET loc_age = ST_makepoint(ST_X(location), ST_Y(location),
Extract('year' FROM birthdate)/100000);
CREATE INDEX users_loc_age_gix ON users USING GIST (loc_age
gist_geometry_ops_nd);
tantanapp.com 24

Updated query
SELECT * FROM users
WHERE loc_age &&&
ST_MakeLine(
ST_MakePoint(180, 90, Extract('year' FROM Now() - interval '74
years')/100000),
ST_MakePoint(-180, -90, Extract('year' FROM Now() - interval '75
years')/100000))
ORDER BY loc_age <<->> ST_MakePoint(103.8, 1.3, 0)
LIMIT 10;
tantanapp.com 25

Looking at the execution plan we
can see that all is good
Limit (cost=0.41..8.43 rows=1 width=136) (actual time=15.294..16.082
rows=10 loops=1)
-> Index Scan using users_loc_age_gix on users (cost=0.41..8.43 rows=1
width=136) (actual time=14.685..14.948 rows=10 loops=1)
Index Cond: (loc_age &&& 'XXX'::geometry)
Order By: (loc_age <<->> 'XXX'::geometry)
Planning time: 0.332 ms
Execution time: 19.053 ms
tantanapp.com 26

Looking at the result we see that
something is wrong
UserID Age
6827677 74 years 11 mons 7 days
1281456 75 years 15 days
5214330 74 years 10 days
6390900 74 years 21 days
tantanapp.com 27

Solution: Keep the non-geometry
where statement
SELECT * FROM users
WHERE age(birthdate) BETWEEN '74 years' AND '75 years'
AND loc_age &&&
ST_MakeLine(
ST_MakePoint(180, 90, Extract('year' FROM Now() - interval '74
years')/100000),
ST_MakePoint(-180, -90, Extract('year' FROM Now() - interval '75
years')/100000))
ORDER BY loc_age <<->> ST_MakePoint(103.8, 1.3, 0)
LIMIT 10;
tantanapp.com 28

Finally we look at Activity
1. Popularity
2. Age
3. Activity <--
tantanapp.com 29

Order by last active time
WITH x AS (SELECT * FROM users
LIMIT 100)
SELECT * FROM x
ORDER BY
ST_Distance(location::geography, ST_MakePoint(103.8, 1.3))
* Extract(Now() - active_time)
LIMIT 10;
tantanapp.com 30

What is the difference between
time, popularity and age?
• Popularity is bounded, a users popularity can range
between 0 and 1.
• Age is also bounded, and static. All our users are
between 16 and 100 years old and their birthdate
never change.
• Time increase infinitely in one direction.
tantanapp.com 31

Adding time to the geo column
ALTER TABLE users ADD COLUMN loc_active geometry;
UPDATE users
SET loc_active = ST_MakePoint(ST_X(location), ST_Y(location),
Extract('epoch' FROM active_time) / 60 / 10000);
CREATE INDEX users_loc_active_gix
ON users USING GIST (loc_active gist_geometry_ops_nd);
tantanapp.com 32

Select with time becomes
SELECT * FROM users
ORDER BY loc_active <<->> ST_MakePoint(103.8, 1.3,
Extract('epoch' FROM Now()) / 60 / 10000)
LIMIT 10;
tantanapp.com 33

What we talked about so far
1. Location X and Y
2. Popularity
3. Age
4. Activity
tantanapp.com 34

Adding a 4th dimension
ALTER TABLE users ADD COLUMN loc_pop_age geometry;
UPDATE users
SET loc_pop_age =
ST_MakePoint(ST_X(location), ST_Y(location),
0.01 * 1 / (popularity+1),
Extract('year' FROM birthdate)/100000);
CREATE INDEX users_loc_pop_age_gix
ON users USING GIST (loc_pop_age gist_geometry_ops_nd);
tantanapp.com 35

Adding a 4th dimension - Query
SELECT * FROM users
WHERE loc_pop_age &&&
ST_MakeLine(
ST_MakePoint(180, 90, -100, Extract('year' FROM Now() - interval
'20 years')/100000),
ST_MakePoint(-180, -90, 100, Extract('year' FROM Now() - interval
'30 years')/100000))
ORDER BY loc_pop_age <<->> ST_MakePoint(103.8, 1.3, 0, 0)
LIMIT 10;
tantanapp.com 36

Query runtimes 2D vs 3D vs 4D
tantanapp.com 37
0 10 20 30 40 50
Location
Location + popularity
Location + age (20-30)
Location + age (74)
Location + activity
Location + pop + age (4D)
3D/4D index (ms) 2D index (ms)

Query times did not impress.
What about index sizes?
tantanapp.com 38
0 50 100 150 200
users_pkey
users_location_gix
users_loc_pop_gix
users_loc_pop_age_gix
users_loc_age_gix
users_loc_active_gix
Size bloated (MB) Size fresh(MB)

Conclusion
1. Popularity
• Pro: Improves ranking?
• Con: Makes it more difficult to reason about the ranking formula
• Con: Makes the query slower
2. Age
• Pro: Fixes query time of outlier queries
• Con: Make the average query time longer
• Con: Have to take special care to not disturb the normal ranking
3. Time
• Pro: Improves ranking?
• Con: Much more difficult to reason about the ranking formula
• Con: Makes the query slower
tantanapp.com 39

Conclusion
Its more difficult to use the 3rd and 4th dimension
than what a quick glance reveals. Test extensively!
tantanapp.com 40

tantanapp.com 42
Thank You!
blomqvist@tantanapp.com
vb@viblo.se

(Ab)using 4d Indexing

Recommended

More Related Content

Viewers also liked (20)

Similar to (Ab)using 4d Indexing (20)

More from PGConf APAC (17)

Recently uploaded (20)

(Ab)using 4d Indexing