MySQL 8.0: Common Table Expressions

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
MySQL 8.0:
Common Table Expressions
Øystein Grøvlen – Senior Principal Software Engineer
MySQL Optimizer Team, Oracle

Program Agenda
Non-recursive common table expressions
Recursive common table expressions
1
2
2

Program Agenda
1
2
3

Common Table Expression (MySQL 8.0.0 labs release)
• A derived table is a subquery in the FROM clause
SELECT … FROM (subquery) AS derived, t1 ...
• Common Table Expression (CTE) is just like a derived table, but its
declaration is put before the query block instead of in FROM clause
WITH derived AS (subquery)
SELECT … FROM derived, t1 ...
• A CTE may precede SELECT/UPDATE/DELETE including sub-queries
WITH derived AS (subquery)
DELETE FROM t1 WHERE t1.a IN (SELECT b FROM derived);
4
Alternative to derived table

CTE
WITH cte_name [( <list of column names> )] AS
(
SELECT ... # Definition
)
[, <any number of other CTE definitions> ]
<SELECT/UPDATE/DELETE statement>
5
Syntax

Common Table Expression (CTE)
WITH qn AS (SELECT a FROM t1)
SELECT * from qn;
WITH qn AS (SELECT a+2 AS a, b FROM t1)
UPDATE t1, qn SET t1.a=qn.a + 10 WHERE t1.a - qn.a = 0;
WITH qn(a, b) AS (SELECT a+2, b FROM t2)
DELETE t1 FROM t1, qn WHERE t1.a - qn.a = 0;
INSERT INTO t2
WITH qn AS (SELECT 10*a AS a FROM t1)
SELECT * from qn;
SELECT * FROM t1 WHERE t1.a IN
(WITH cte as (SELECT * FROM t1 AS t2 LIMIT 1)
SELECT a + 0 FROM cte);
6
Examples

Common Table Expression versus Derived Table
Better readability
Can be referenced multiple times
Can refer to other CTEs
Improved performance
7

Better readability
• Derived table:
SELECT …
FROM t1 LEFT JOIN ((SELECT … FROM …) AS dt JOIN t2 ON …) ON …
• CTE:
WITH dt AS (SELECT ... FROM ...)
SELECT ...
FROM t1 LEFT JOIN (dt JOIN t2 ON ...) ON ...
8

Can be referenced multiple times
• Derived table can not be referenced twice:
SELECT ...
FROM (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d1
JOIN (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b) AS d2 ON d1.b = d2.a;
• CTE can:
WITH d AS (SELECT a, b, SUM(c) s FROM t1 GROUP BY a, b)
SELECT ... FROM d AS d1 JOIN d AS d2 ON d1.b = d2.a;
9

Can refer to other CTEs
• Derived tables can not refer to other derived tables:
SELECT …
FROM (SELECT … FROM …) AS d1, (SELECT … FROM d1 …) AS d2 …
ERROR: 1146 (42S02): Table ‘db.d1’ doesn’t exist
• CTEs can refer other CTEs:
WITH d1 AS (SELECT … FROM …),
d2 AS (SELECT … FROM d1 …)
SELECT
FROM d1, d2 …
10

Chained CTEs
WITH cte1(txt) AS (SELECT "This "),
cte2(txt) AS (SELECT CONCAT(cte1.txt,"is a ") FROM cte1),
cte3(txt) AS (SELECT "nice query" UNION
SELECT "query that rocks" UNION
SELECT "query"),
cte4(txt) AS (SELECT concat(cte2.txt, cte3.txt) FROM cte2, cte3)
SELECT MAX(txt), MIN(txt) FROM cte4;
+----------------------------+----------------------+
| MAX(txt) | MIN(txt) |
+----------------------------+----------------------+
| This is a query that rocks | This is a nice query |
+----------------------------+----------------------+
1 row in set (0,00 sec)
11
Neat, but not very useful example

Better performance
• Derived table:
– For derived tables that are materialized, two identical derived tables will be
materialized. Performance problem (more space, more time, longer locks)
– Similar with view references
• CTE:
– Will be materialized once, regardless of how many references
12

DBT3 Query 15
CREATE VIEW revenue0 (supplier_no, total_revenue)
AS SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount))
FROM lineitem
WHERE l_shipdate >= '1996-07-01'
AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day)
GROUP BY l_suppkey;
SELECT s_suppkey, s_name, s_address, s_phone, total_revenue
FROM supplier, revenue0
WHERE s_suppkey = supplier_no
AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0)
ORDER BY s_suppkey;
Top Supplier Query

Materialization of view
DBT-3 Query 15
ORDER JOIN
Supplier
(eq_ref)
GROUP Lineitem
(range)
Materialize
GROUP Lineitem
(range)
Materialize
SELECT
Subquery
Revenue0
Revenue0

DBT-3 Query 15
EXPLAIN
CTE
id
select
type
table type possible keys key rows filtered Extra
1 PRIMARY <derived3> ALL NULL NULL 4801074 10.00
Using where;
Using temporary;
Using filesort
1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL
3 DERIVED lineitem range i_l_shipdate, ... i_l_shipdate 4801074 100.00 Using temporary
2 SUBQUERY <derived4> ALL NULL NULL 4801074 100.00 NULL

CTE Materialization
WITH revenue0 (supplier_no, total_revenue) AS
(SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount))
FROM lineitem
WHERE l_shipdate >= '1996-07-01'
AND l_shipdate < DATE_ADD('1996-07-01‘, INTERVAL '90' day)
GROUP BY l_suppkey)
SELECT s_suppkey, s_name, s_address, s_phone, total_revenue
FROM supplier, revenue0
WHERE s_suppkey = supplier_no
AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0)
ORDER BY s_suppkey;
DBT3 Query 15: Top Supplier Query

DBT-3 Query 15, CTE
EXPLAIN
CTE
id
select
type
table type possible keys key rows filtered Extra
1 PRIMARY <derived2> ALL NULL NULL 4801074 10.00
Using where;
Using temporary;
Using filesort
1 PRIMARY supplier eq_ref PRIMARY PRIMARY 1 100.00 NULL
3 SUBQUERY <derived2> ALL NULL NULL 4801074 100.00 NULL

DBT-3 Query 15
Materialization of CTE
ORDER JOIN
Supplier
(eq_ref)
GROUP Lineitem
(range)
Materialize
SELECT
Subquery
Revenue0
FROM

DBT-3 Query 15
Confidential – Oracle Internal/Restricted/Highly Restricted 19
Query Performance
0
2
4
6
8
10
12
14
16
18
View CTE
QueryExecutionTime(seconds)

Non-recursive CTE
• Find best and worst month:
WITH sales_by_month(month,total) AS
# first CTE: one row per month, with amount sold on all days of month
(SELECT MONTH(day_of_sale), SUM(amount) FROM sales_days
WHERE YEAR(day_of_sale)=2015
GROUP BY MONTH(day_of_sale)),
best_month(month, total, award) AS # second CTE: best month
(SELECT month, total, "best" FROM sales_by_month
WHERE total=(SELECT MAX(total) FROM sales_by_month)),
worst_month(month, total, award) AS # 3rd CTE: worst month
(SELECT month, total, "worst" FROM sales_by_month
WHERE total=(SELECT MIN(total) FROM sales_by_month))
# Now show best and worst:
SELECT * FROM best_month UNION All SELECT * FROM worst_month;
20
A more useful example

Non-recursive CTE
• Result:
+-------+-------+-------+
| month | total | award |
+-------+-------+-------+
| 1 | 300 | best |
| 3 | 11 | worst |
+-------+-------+-------+
21
A more useful example

Program Agenda
1
2
22

Recursive CTE
• A recursive CTE refers to itself in a subquery
• The “seed” SELECT is executed once to create the initial data subset, the
recursive SELECT is repeatedly executed to return subsets of data until the
complete result set is obtained.
• Recursion stops when an iteration does not generate any new rows
• Useful to dig in hierarchies (parent/child, part/subpart)
23
WITH RECURSIVE cte AS
( SELECT ... FROM table_name /* "seed" SELECT */
UNION ALL
SELECT ... FROM cte, table_name) /* "recursive" SELECT */
SELECT ... FROM cte;

Recursive CTE
24
A simple example
Print 1 to 10 :
WITH RECURSIVE qn AS
( SELECT 1 AS a
UNION ALL
SELECT 1+a FROM qn WHERE a<10
)
SELECT * FROM qn;
a
1
2
3
4
5
6
7
8
9
10

Recursive CTE
25
INSERT
Insert 1 to 10 :
INSERT INTO numbers
WITH RECURSIVE qn AS
( SELECT 1 AS a
UNION ALL
SELECT 1+a FROM qn WHERE a<10
)
SELECT * FROM qn;
SELECT * FROM numbers;
a
1
2
3
4
5
6
7
8
9
10

Date sequence
26
Missing dates
SELECT orderdate,
SUM(totalprice) sales
FROM orders
GROUP BY orderdate
ORDER BY orderdate;
+------------+-----------+
| orderdate | sales |
+------------+-----------+
| 2016-09-01 | 43129.83 |
| 2016-09-03 | 218347.61 |
| 2016-09-04 | 142568.40 |
| 2016-09-05 | 299244.83 |
| 2016-09-07 | 185991.79 |
+------------+-----------+

Date sequence
27
All dates
WITH RECURSIVE dates(date) AS
( SELECT '2016-09-01'
UNION ALL
SELECT DATE_ADD(date, INTERVAL 1 DAY)
FROM dates
WHERE date < '2016-09-07‘ )
SELECT dates.date,
COALESCE(SUM(totalprice), 0) sales
FROM dates LEFT JOIN orders
ON dates.date = orders.orderdate
GROUP BY dates.date
ORDER BY dates.date;
+------------+-----------+
| date | sales |
+------------+-----------+
| 2016-09-01 | 43129.83 |
| 2016-09-02 | 0.00 |
| 2016-09-03 | 218347.61 |
| 2016-09-04 | 142568.40 |
| 2016-09-05 | 299244.83 |
| 2016-09-06 | 0.00 |
| 2016-09-07 | 185991.79 |
+------------+-----------+

n un unp 1
1 1 1
2 1 2
3 2 3
4 3 5
5 5 8
6 8 13
7 13 21
8 21 34
9 34 55
10 55 89
Fibonacci Numbers
28
WITH RECURSIVE qn(n, un, unp1) AS
( SELECT 1, 1 , 1
UNION ALL
SELECT 1+n, unp1, un+unp1
FROM qn WHERE n<10)
SELECT * FROM qn;

Hierarchy Traversal
29
Employee database
CREATE TABLE employees (
id INT PRIMARY KEY,
name VARCHAR(100),
manager_id INT,
FOREIGN KEY (manager_id)
REFERENCES employees(id) );
INSERT INTO employees VALUES
(333, "Yasmina", NULL), # CEO
(198, "John", 333), # John reports to 333
(692, "Tarek", 333),
(29, "Pedro", 198),
(4610, "Sarah", 29),
(72, "Pierre", 29),
(123, "Adil", 692);

Hierarchy Traversal
30
List reporting chain
WITH RECURSIVE
emp_ext (id, name, path) AS (
SELECT id, name, CAST(id AS CHAR(200))
FROM employees
WHERE manager_id IS NULL
UNION ALL
SELECT s.id, s.name,
CONCAT(m.path, ",", s.id)
FROM emp_ext m JOIN employees s
ON m.id=s.manager_id )
SELECT * FROM emp_ext ORDER BY path;
id name path
333 Yasmina 333
198 John 333,198
692 Tarek 333,692
29 Pedro 333,198,29
123 Adil 333,692,123
4610 Sarah 333,198,29,4610
72 Pierre 333,198,29,72

Hierarchy Traversal
31
List descendants
WITH RECURSIVE
emp_ext (id, name, path) AS (
SELECT id, name, CAST(id AS CHAR(200))
FROM employees
WHERE manager_id IS NULL
UNION ALL
SELECT s.id, s.name,
CONCAT(m.path, ",", s.id)
FROM emp_ext m JOIN employees s
ON m.id=s.manager_id )
SELECT * FROM emp_ext ORDER BY path;
id name path
333 Yasmina 333
198 John 333,198
29 Pedro 333,198,29
4610 Sarah 333,198,29,4610
72 Pierre 333,198,29,72
692 Tarek 333,692
123 Adil 333,692,123

Recursive CTE
• WITH RECURSIVE cte_name [list of column names ] AS
(
SELECT ... <-- specifies initial set
UNION ALL
SELECT ... <-- specifies initial set
UNION ALL
...
SELECT ... <-- specifies how to derive new rows
UNION ALL
SELECT ... <-- specifies how to derive new rows
...
)
[, any number of other CTE definitions ]
32
Complete syntax

Recursive CTE
• A recursive SELECT must not contain
– GROUP BY
– Aggregate functions (like SUM)
– ORDER BY
– LIMIT
– DISTINCT
• A recursive SELECT must reference the CTE only once and only in its FROM
clause, not in any subquery
• A recursive SELECT can not be the inner table of an outer join
Restrictions

Want to try it out?
• Goto labs.mysql.com
• Select MySQL Server 8.0.0 Optimizer
• Available as
– Linux binaries
– Source code
• Warning!
– For testing purposes only!
– NOT FIT FOR PRODUCTION.
34

Other New Features in 8.0.0 Optimizer Labs Release
• MERGE() and NO_MERGE() hints
WITH cte AS ( ... ) SELECT /*+ NO_MERGE(cte) */ ... FROM cte ...
• Column name aliases for derived tables
SELECT ... FROM (SELECT ...) dt(a, b, c) ...
• Descending index
• Join order hints
SELECT /*+ JOIN_ORDER(t1, t2) */ * FROM t1 JOIN t2 ...
• JSON aggregation function
– JSON_ARRAYAGG
– JSON_OBJECTAGG

Want to learn more?
• MySQL Server Team blog
– http://mysqlserverteam.com/
– http://mysqlserverteam.com/mysql-8-0-labs-recursive-common-table-expressions-in-
mysql-ctes/
– More to come
• My blog:
– http://oysteing.blogspot.com/
• MySQL forum
– Optimizer & Parser: http://forums.mysql.com/list.php?115

MySQL 8.0: Common Table Expressions

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to MySQL 8.0: Common Table Expressions (20)

Recently uploaded (20)

MySQL 8.0: Common Table Expressions