Looking Inside the MySQL 8.0 Document Store

Looking Inside the MySQL
Document Store
A JSON Document Store with all the benefits
of a traditional RDBMS (DEV2902)
Frédéric Descamps
Community Manager
MySQL
September 2019
2 / 128

Safe Harbor
The following is intended to outline our general product direction. It is intended for information purpose only, and
may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any
features or functionality described for Oracle's product may change and remains at the sole discretion of Oracle
Corporation.
Statement in this presentation relating to Oracle's future plans, expectations, beliefs, intentions and ptospects are
"forward-looking statements" and are subject to material risks and uncertainties. A detailed discussion of these
factors and other risks that a ect our business is contained in Oracle's Securities and Exchange Commission
(SEC) lings, including our most recent reports on Form 10-K and Form 10-Q under the heading "Risk Factors".
These lings are available on the SEC's website or on Oracle's website at h p://www.oracle.com/investor. All
information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any
statement in light of new information or future events.
Copyright @ 2019 Oracle and/or its affiliates.
3 / 128

about.me/lefred
Who am I ?
4 / 128

Frédéric Descamps
@lefred
MySQL Evangelist
Managing MySQL since 3.23
devops believer
living in Belgium 🇧🇪
h ps://lefred.be
5 / 128

MySQL 8.0
a Document Store with all the benefits of
a RDBMS
6 / 128

Why ?
7 / 128

Why ?
Developers don't really like SQL anymore...

8 / 128

Why ?
SQL can be complicated and slows down the initial development
9 / 128

easy operations
What do developers want ?
10 / 128

Use Objects / Documents
Developers want just to use objects
11 / 128

that's why they usually love what DBAs hate the most ORMs !).
12 / 128

They want to deal with these objects easily (CRUD operations) and they don't want to think
about schema design (slows down the initial development process).
13 / 128

They want to deal with these objects easily (CRUD operations) and they don't want to think
about schema design (slows down the initial development process).
But they also want to keep their data safe and use transactions.
14 / 128

MySQL as RDBMS
Relational Databases
15 / 128

Data Integrity
16 / 128

Data Integrity
normalization
17 / 128

Data Integrity
normalization
constraints (foreign keys, ...)
18 / 128

Data Integrity
normalization
Atomicity, Consistency, Isolation, Durability
19 / 128

Data Integrity
normalization
ACID compliant
20 / 128

Data Integrity
normalization
ACID compliant
transactions
21 / 128

Data Integrity
normalization
ACID compliant
transactions
SQL
22 / 128

Data Integrity
normalization
ACID compliant
transactions
SQL
powerfull query language
23 / 128

MySQL as Document Store
NoSQL Databases
24 / 128

NoSQL or Document Store
Schemaless
25 / 128

Schemaless
no schema design, no normalization, no foreign keys, no data types, ...
26 / 128

Schemaless
very quick initial development
27 / 128

Schemaless
Flexible data structure
28 / 128

Schemaless
embedded arrays or objects
29 / 128

Schemaless
valid solution when natural data can't be modelized optimaly into a relational model
30 / 128

Schemaless
objects persistance without the use of any ORM - mapping oobject-oriented
31 / 128

Schemaless
JSON
32 / 128

Schemaless
JSON
close to frontend
33 / 128

Schemaless
JSON
close to frontend
easy to learn
34 / 128

How DBAs see data
35 / 128

How DBAs see data How Developers see
data
{
"GNP" : 249704,
"Name" : "Belgium",
"government" : {
"GovernmentForm" :
"Constitutional Monarchy, Federation",
"HeadOfState" : "Philippe I"
},
"_id" : "BEL",
"IndepYear" : 1830,
"demographics" : {
"Population" : 10239000,
"LifeExpectancy" : 77.8000030517578
},
"geography" : {
"Region" : "Western Europe",
"SurfaceArea" : 30518,
"Continent" : "Europe"
}
}
36 / 128

What if there was a way to provide both SQL
and NoSQL on one stable platform that has
proven stability on well know technology with a
large Community and a diverse ecosystem ?
37 / 128

DBMS or NoSQL ?
38 / 128

DBMS or NoSQL ?
Why not both ?
39 / 128

The MySQL Document Store !
SQL is now optional ?!
40 / 128

SQL is now optional ?!
41 / 128

Developers:
[x] schemaless
[x] rapid prototying/simpler APIs
[x] document model
[x] transactions
Operations:
[x] performance
management/visibility
[x] robust replication, backup, restore
[x] comprehensive tooling ecosystem
[x] simpler application schema
upgrades
Business Owner:
[x] don't lose my data == ACID trx
[x] capture all my data = extensible/schemaless
[x] product on schedule/time to market = rapid
developement
A solution for all
42 / 128

MySQL Document Store
the Solution
43 / 128

Built on
the MySQL
JSON Data
type and
Proven
MySQL
Server
Technology
Provides a schema exible JSON Document Store
No SQL required
No need to de ne all possible a ributes, tables, etc.
Uses new X DevAPI
Can leverage generated column to extract JSON values into
materialized columns that can be indexed for fast SQL searches.
Document can be ~1GB
It's a column in a row of a table
It cannot exceed max_allowed_packet
Allows use of modern programming styles
No more embedded strings of SQL in your code
Easy to read
Also works with relational Tables
Proven MySQL Technology
44 / 128

X Protocol Connectors
45 / 128

X DevAPI We provide connectors for
C++, Java, .Net, Node.js, Python, PHP
working with Communities to help them supporting it too
New MySQL Shell
Command Completion
Python, JavaScrips & SQL modes
Admin functions
New Util object
A new high-level session concept that can scale from single
MySQL Server to a multiple server environment
Non-blocking, asynchronous calls follow common language
pa erns
Supports CRUD operations
46 / 128

PY
JS
SQL
>
MySQL 8.0
Upgrade Checker
Auto Completion
&
Command History
Output
Formats
(table, json, tabs)
Prompt
Themes
Batch
Execution
5.7
8.0
JS
Document
Store
X dev API
SQL CLI InnoDB
Cluster
importJSON
JSJS
47 / 128

Setup
48 / 128

Installing MySQL Document Store
install MySQL 8.0
49 / 128

install MySQL 8.0
install MySQL Shell
50 / 128

install MySQL 8.0
install MySQL Shell
install MySQL Connector for your programming language
51 / 128

install MySQL 8.0
install MySQL Shell
php-pecl-mysql-xdevapi for PHP
mysql-connector-python for Python
...
52 / 128

install MySQL 8.0
install MySQL Shell
php-pecl-mysql-xdevapi for PHP
mysql-connector-python for Python
...
And nothing else, no need to install anything else or load any plugin, just be sure your
rewall allows you to connect through port 33060 (X Protocol).
53 / 128

Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
54 / 128

Migration from MongoDB to MySQL DS
For this example, I will use the well known restaurants collection:
55 / 128

56 / 128

57 / 128

Let's make a query
58 / 128

Let's make a query
That's too much records to show in here... let's limit it
59 / 128

60 / 128

Some more examples
61 / 128

Let's add a selection criteria:
62 / 128

Using IN...
63 / 128

Syntax slightly different than MongoDB
64 / 128

Syntax slightly different than MongoDB
65 / 128

And for developers ?
66 / 128

$session = mysql_xdevapigetSession("mysqlx://fred:MyP@ssw0rd%@localhost");
$schema = $session->getSchema("docstore");
$collection = $schema->getCollection("restaurants");
$results = $collection-> nd($search)->execute()->fetchAll();
...
foreach ($results as $doc) {
echo "<tr><td><a href='?id=${doc[_id]}'>${doc[name]}</a></td>";
echo "<td>${doc[borough]}</td><td>${doc[cuisine]}</td></tr>";
}
67 / 128

...
}

Easy, using only CRUD operations !
68 / 128

...
}

Easy, using only CRUD operations !
Not a single SQL statement !
69 / 128

CRUD operations
70 / 128

CRUD operations for collections
Add a document
71 / 128

Add a document
collection.add({ name: 'fred', age: 42 })
.add({ name: 'dave', age: 23 })
.execute()
collection.add([
{ name: 'dimo', age: 50 },
{ name: 'kenny', age: 25 }
]).execute()
72 / 128

Modify a document
73 / 128

Modify a document
collection.modify('name = :name')
.bind('name', 'fred')
.set('age', 43)
.sort('name ASC')
.limit(1)
.execute()
collection.modify('name = :name')
.patch({ age: 43, active: false })
.sort('name DESC')
.limit(1)
.execute()
74 / 128

Remove a document
75 / 128

Remove a document
collection.remove('name = :name')
.sort('age ASC')
.limit(1)
.execute()
76 / 128

MySQL Document Store Objects Summary
77 / 128

All you need to know is here:
h ps://dev.mysql.com/doc/x-devapi-userguide/en/crud-operations-overview.html
78 / 128

we do care about your data
MySQL Document Store is Full ACID
Compliant
79 / 128

Document Store Full ACID !
It relies on the proven MySQL InnoDB's strength & robustness:
80 / 128

innodb_ ush_log_at_trx_commit = 1
81 / 128

innodb_doublewrite = ON
82 / 128

sync_binlog = 1
83 / 128

sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
84 / 128

sync_binlog = 1
transaction_isolation = REPEATABLE-READ|READ-COMMITTED|...
We do care about your data !
85 / 128

MySQL DS Full ACID - Transactions support
86 / 128

MySQL DS Full ACID - Transactions support
87 / 128

OK we have Document Store, CRUD and ACID
but what makes MySQL Document Store
unique ?
88 / 128

Challenge: list the best restaurant of each type
of food and show the top 10, with the best one
first !

don't forget that all these restaurants are just JSON documents
89 / 128

NoSQL as SQL - aggregation
90 / 128

91 / 128

92 / 128

NoSQL or SQL
You have the possibility to write clean and neat code:
93 / 128

NoSQL or SQL
$results = $collection-> nd('cuisine like "italian"')->execute()->fetchAll();
94 / 128

NoSQL or SQL
Or use SQL only when it's really needed:
95 / 128

NoSQL or SQL
$results = $session->sql('WITH cte1 AS (SELECT doc->>"$.name" AS name,
doc->>"$.cuisine" AS cuisine,
(SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT
PATH "$.score")) AS r) AS avg_score FROM docstore.restaurants)
SELECT *, RANK()
OVER ( PARTITION BY cuisine ORDER BY avg_score) AS `rank`
FROM cte1 ORDER BY `rank`, avg_score DESC LIMIT 10;')->execute();
96 / 128

NoSQL or SQL
$results = $session->sql('WITH cte1 AS (SELECT doc->>"$.name" AS name,
doc->>"$.cuisine" AS cuisine,
(SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT
PATH "$.score")) AS r) AS avg_score FROM docstore.restaurants)
SELECT *, RANK()
OVER ( PARTITION BY cuisine ORDER BY avg_score) AS `rank`
FROM cte1 ORDER BY `rank`, avg_score DESC LIMIT 10;')->execute();
All in the same MySQL X Session !
97 / 128

You can mix NoSQL & SQL as you want
98 / 128

Best of Both Worlds: JSON_TABLE
What are the maximum 10 ratings ever given to a restaurant?
99 / 128

Best of Both Worlds: JSON_TABLE
What are the maximum 10 ratings ever given to a restaurant?
Cool... but my app only process JSON !
100 / 128

Best of Both Worlds: JSON_TABLE (2)
With JSON output:
101 / 128

And the DBA ?
102 / 128

Role of the MySQL Document Store DBA ?
103 / 128

nd the non optimal lookups
create the necessary indexes based on
the results

Role of the MySQL Document Store DBA ?
104 / 128

Role of the MySQL Document Store DBA
Looking for full table scans:
SELECT schema_name, sum_rows_examined, (sum_rows_examined/exec_count) avg_rows_call,
format_time(total_latency) tot_lat, exec_count,
format_time(total_latency/exec_count) AS latency_per_call, query_sample_text
FROM sys.x$statements_with_full_table_scans AS t1
JOIN performance_schema.events_statements_summary_by_digest AS t2
ON t2.digest=t1.digest
WHERE schema_name = "docstore" AND query_sample_text LIKE '%select%'
ORDER BY (total_latency/exec_count) descG
105 / 128

Role of the MySQL Document Store DBA
Looking for full table scans:
SELECT schema_name, sum_rows_examined, (sum_rows_examined/exec_count) avg_rows_call,
format_time(total_latency) tot_lat, exec_count,
format_time(total_latency/exec_count) AS latency_per_call, query_sample_text
FROM sys.x$statements_with_full_table_scans AS t1
JOIN performance_schema.events_statements_summary_by_digest AS t2
ON t2.digest=t1.digest
WHERE schema_name = "docstore" AND query_sample_text LIKE '%select%'
ORDER BY (total_latency/exec_count) descG
More info at h ps://lefred.be/content/mysql-8-0-if-i-should-optimize-only-one-query-
on-my-application-which-one-should-it-be/
106 / 128

Role of the MySQL Document Store DBA (2)
107 / 128

108 / 128

Getting the Query Explain Plan:
109 / 128

Getting the Query Explain Plan:
It's indeed a full table scan ! What should we do ?
110 / 128

Create Indexes:
111 / 128

Create Indexes:
For the "old" experienced MySQL DBAs, you can instead use:
112 / 128

INFO: JSON Notation Shortcut
113 / 128

Result:
114 / 128

For the curious: a collection in SQL:
115 / 128

NEW in MySQL 8.0.17
Multi-valued indexes
JSON_CONTAINS()
MEMBER OF() new
JSON_OVERLAPS() new
JSON Schema
JSON_SCHEMA_VALID(<json schema>,<json doc>) new
JSON_SCHEMA_VALIDATION_REPORT(<json schema>,<json doc>) new
116 / 128

JSON Multi-Valued Indexes 8.0.17
117 / 128

JSON Multi-Valued Indexes 8.0.17 (2)
118 / 128

JSON Schema Validation 8.0.17
119 / 128

JSON Schema Validation (2) 8.0.17
120 / 128

De ne the validation as a constraint:
121 / 128

De ne the validation as a constraint:
And the result in action:
122 / 128

what do I gain ?
Conclusion
123 / 128

schemaless
exible data structure
easy to start (CRUD)
Conclusion
This is the best of the two worlds in one product !
Data integrity
ACID Compliant
Transactions
SQL
124 / 128

http://lefred.be/content/top-10-reasons-for-
nosql-with-mysql/
125 / 128

Thank you !
127 / 128

Session Survey
Help us make the content even be er.
Please complete the session survey in
the Mobile App.
128 / 128

Looking Inside the MySQL 8.0 Document Store

Recommended

More Related Content

What's hot (20)

Similar to Looking Inside the MySQL 8.0 Document Store (20)

More from Frederic Descamps (20)

Recently uploaded (20)

Looking Inside the MySQL 8.0 Document Store