![Node.JS 27
// Connecting to MySQL Server and working with a Collection
var mysqlx = require('@mysql/xdevapi');
var db;
// Connect to server
mysqlx
.getSession({
user: 'user',
password: 'password',
host: 'localhost',
port: '33060',
})
.then(function (session) {
db = session.getSchema('test');
// Create a new collection 'my_collection'
return db.createCollection('my_collection');
})
.then(function (myColl) {
// Insert documents
return Promise
.all([
myColl.add({ name: 'Sakila', age: 15 }).execute(),
myColl.add({ name: 'Susanne', age: 24 }).execute(),
myColl.add({ name: 'User', age: 39 }).execute()
])
.then(function () {
// Find a document
return myColl
.find('name like :name && age < :age')
.bind({ name: 'S%', age: 20 })
.limit(1)
.execute(function (doc) {
// Print document
console.log(doc);
});
});
})
.then(function(docs) {
// Drop the collection
return db.dropCollection('my_collection');
})
.catch(function(err) {
// Handle error
});](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-27-320.jpg)
![Dump and load using MySQL Shell & Python
This example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo quiet eval 'DBQuery.shellBatchSize=30000;
db.restaurants.find().shellPrint()'
| perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json
37](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-37-320.jpg)
![42
More Examples!
restaurants.find().fields([“name”,”cuisine”]).limit(2
)](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-42-320.jpg)
![43
Comparing Syntax: MongoDB vs MYSQL
MongoDB:
> db.restaurants.find({"cuisine": "French",
"borough": { $not: /^Manhattan/} },
{"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2)
MySQL:
>restaurants.find(“cuisine=’French’ AND
borough!=’Manhattan’”).fields([“name”,”cuisine”,”borough”
]).limit(2)](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-43-320.jpg)
![[
{
"date": {
"$date": 1416009600000
},
"grade": "Z",
"score": 38
},
{
"date": {
"$date": 1398988800000
},
"grade": "A",
"score": 10
},
{
"date": {
"$date": 1362182400000
},
"grade": "A",
"score": 7
},
{
"date": {
"$date": 1328832000000
},
"grade": "A",
"score": 13
}
] 67
Embedded
Arrays of
values can be
messy to
traverse.](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-67-320.jpg)
![$.grades[0]
$.greades[1 to 2]
$.grades[first]
$.grades[last]
$.grades[first to last - 1]
69
Arrays are now simple](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-69-320.jpg)
![Find the top 10 restaurants by grade for each cuisine 73
WITH cte1 AS
(SELECT doc->>"$.name" AS name,
doc->>"$.cuisine" AS cuisine,
(SELECT AVG(score) FROM
JSON_TABLE(doc, "$.grades[*]" COLUMNS
(score INT PATH "$.score")) AS r) AS avg_score
FROM restaurants)
SELECT *, RANK() OVER
(PARTITION BY cuisine ORDER BY avg_score DESC) AS `rank`
FROM cte1
ORDER BY `rank`, avg_score DESC LIMIT 10;
This query uses a Common Table Expression (CTE) and a Windowing Function to rank the
average scores of each restaurant, by each cuisine assembled in a JSON_TABLE](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-73-320.jpg)
![GeoJSON support too!
mysql> SELECT ST_AsGeoJSON(ST_GeomFromText('POINT(11.11111
12.22222)'),2);
+-------------------------------------------------------------+
| ST_AsGeoJSON(ST_GeomFromText('POINT(11.11111 12.22222)'),2) |
+-------------------------------------------------------------+
| {"type": "Point", "coordinates": [11.11, 12.22]} |
+-------------------------------------------------------------+
79](https://image.slidesharecdn.com/mysqldocumentstore-190313152636/85/MySQL-Without-the-SQL-Oh-My-MySQL-Document-Store-Confoo-CA-2019-79-320.jpg)
MySQL Without the SQL - Oh My! -> MySQL Document Store -- Confoo.CA 2019
- 1. MySQL Document Store - A NoSQL JSON Document Database Dave Stokes @stoker [email protected] Elephantdolphin.blogger.com OpensourceDBA.wordpress.com
- 2. MySQL Without the SQL - Oh My! Dave Stokes @stoker [email protected] Elephantdolphin.blogger.com OpensourceDBA.wordpress.com
- 3. 3
- 4. Safe Harbor Agreement THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. THE DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES OR FUNCTIONALITY DESCRIBED FOR ORACLE'S PRODUCTS REMAINS AT THE SOLE DISCRETION OF ORACLE. 4
- 8. Relational Databases ● Data Integrity ○ Normalization ○ constraints (foreign keys, ...) ● Atomicity, Consistency, Isolation, Durability ○ ACID compliant ○ transactions ● SQL ○ powerful query language 8
- 9. Relational Databases ● Need to set up tables BEFORE use ● Relations, indexes, data normalization, query optimizations ● Hard to change on the fly ● Need a DBA or someone who has DBA skills 9
- 10. NoSQL or Document Store 10
- 11. NoSQL or Document Store ● Schemaless ○ No schema design, no normalization, no foreign keys, no data types, … ○ Very quick initial development ● Flexible data structure ○ Embedded arrays or objects ○ Valid solution when natural data can not be modelized optimally into a relational model ○ Objects persistence without the use of any ORM - *mapping object- oriented* 11
- 12. NoSQL or Document Store ● JSON ● close to frontend ● native in JS ● easy to learn 12
- 13. How DBAs see data as opposed to how Developers see data { "GNP" : 249704, "Name" : "Belgium", "government" : { "GovernmentForm" : "Constitutional Monarchy, Federation", "HeadOfState" : "Philippe I" }, "_id" : "BEL", "IndepYear" : 1830, "demographics" : { "Population" : 10239000, "LifeExpectancy" : 77.8000030517578 }, "geography" : { "Region" : "Western Europe", "SurfaceArea" : 30518, "Continent" : "Europe" } } 13
- 14. What if there was a way to provide both SQL and NoSQL on one stable platform that has proven stability on well know technology with a large Community and a diverse ecosystem ? With the MySQL Document Store it is now an option! 14
- 15. A Solution for all Developers: schemaless ★ rapid prototyping & simpler APIs ★ document model ★ transactions Operations: ★ performance management/visibility ★ robust replication, backup, restore ★ comprehensive tooling ecosystem ★ simpler application schema upgrades 15 Business Owner: ★ don't lose my data == ACID trx ★ capture all my data = extensible/schemaless ★ product on schedule/time to market = rapid development
- 16. Built on the MySQL JSON Data type and Proven MySQL Server Technology 16 ★ Provides a schema flexible JSON Document Store ★ No SQL required ★ No need to define all possible attributes, tables, etc. ★ Uses new X DevAPI ★ Can leverage generated column to extract JSON values into materialized columns that can be indexed for fast SQL searches.
- 17. Built on the MySQL JSON Data type and Proven MySQL Server Technology 17 ★ Document can be ~1GB ○ It's a column in a row of a table ★ Allows use of modern programming styles ○ No more embedded strings of SQL in your code ○ Easy to read ★ Also works with relational Tables ★ Proven MySQL Technology
- 18. ★ C++ ★ Java ★ .Net ★ Node.js ★ JavaScript ★ Python ★ PHP ○ Working with other Communities to help them supporting it too 18 Connectors for
- 19. ★ Command Completion ★ Python, JavaScripts & SQL modes ★ Admin functions ★ New Util object ★ A new high-level session concept that can scale from single MySQL Server to a multiple server environment 19 New MySQL Shell
- 20. ★ Non-blocking, asynchronous calls follow common language patterns ★ Supports CRUD operations 20 New Model
- 21. 21 X Protocol built on Google Protobufs
- 22. 22 Architecture of both Old and New Protocols
- 23. 23 How Your Application will work with InnoDB Cluster
- 24. But what does this look like in PHP?? 24
- 25. JavaScript 25 // Connecting to MySQL Server and working with a Collection var mysqlx = require('mysqlx'); // Connect to server var mySession = mysqlx.getSession( { host: 'localhost', port: 33060, user: 'user', password: 'password'} ); var myDb = mySession.getSchema('test'); // Create a new collection 'my_collection' var myColl = myDb.createCollection('my_collection'); // Insert documents myColl.add({_id: '1', name: 'Sakila', age: 15}).execute(); myColl.add({_id: '2', name: 'Susanne', age: 24}).execute(); myColl.add({_id: '3', name: 'User', age: 39}).execute(); // Find a document var docs = myColl.find('name like :param1 AND age < :param2').limit(1). bind('param1','S%').bind('param2',20).execute(); // Print document print(docs.fetchOne()); // Drop the collection myDb.dropCollection('my_collection');
- 26. Python 26 # Connecting to MySQL Server and working with a Collection from mysqlsh import mysqlx # Connect to server mySession = mysqlx.get_session( { 'host': 'localhost', 'port': 33060, 'user': 'user', 'password': 'password'} ) myDb = mySession.get_schema('test') # Create a new collection 'my_collection' myColl = myDb.create_collection('my_collection') # Insert documents myColl.add({'_id': '1', 'name': 'Sakila', 'age': 15}).execute() myColl.add({'_id': '2', 'name': 'Susanne', 'age': 24}).execute() myColl.add({'_id': '3', 'name': 'User', 'age': 39}).execute() # Find a document docs = myColl.find('name like :param1 AND age < :param2') .limit(1) .bind('param1','S%') .bind('param2',20) .execute() # Print document doc = docs.fetch_one() print doc
- 27. Node.JS 27 // Connecting to MySQL Server and working with a Collection var mysqlx = require('@mysql/xdevapi'); var db; // Connect to server mysqlx .getSession({ user: 'user', password: 'password', host: 'localhost', port: '33060', }) .then(function (session) { db = session.getSchema('test'); // Create a new collection 'my_collection' return db.createCollection('my_collection'); }) .then(function (myColl) { // Insert documents return Promise .all([ myColl.add({ name: 'Sakila', age: 15 }).execute(), myColl.add({ name: 'Susanne', age: 24 }).execute(), myColl.add({ name: 'User', age: 39 }).execute() ]) .then(function () { // Find a document return myColl .find('name like :name && age < :age') .bind({ name: 'S%', age: 20 }) .limit(1) .execute(function (doc) { // Print document console.log(doc); }); }); }) .then(function(docs) { // Drop the collection return db.dropCollection('my_collection'); }) .catch(function(err) { // Handle error });
- 28. C++ 28 // Connect to server var mySession = MySQLX.GetSession("server=localhost;port=33060;user=user;password=password;"); var myDb = mySession.GetSchema("test"); // Create a new collection "my_collection" var myColl = myDb.CreateCollection("my_collection"); // Insert documents myColl.Add(new { name = "Sakila", age = 15}).Execute(); myColl.Add(new { name = "Susanne", age = 24}).Execute(); myColl.Add(new { name = "User", age = 39}).Execute(); // Find a document var docs = myColl.Find("name like :param1 AND age < :param2").Limit(1) .Bind("param1", "S%").Bind("param2", 20).Execute(); // Print document Console.WriteLine(docs.FetchOne()); // Drop the collection myDb.DropCollection("my_collection");
- 29. Java 29 // Connect to server Session mySession = new SessionFactory().getSession("mysqlx://localhost:33060/test?user=user&password=password"); Schema myDb = mySession.getSchema("test"); // Create a new collection 'my_collection' Collection myColl = myDb.createCollection("my_collection"); // Insert documents myColl.add("{"name":"Sakila", "age":15}").execute(); myColl.add("{"name":"Susanne", "age":24}").execute(); myColl.add("{"name":"User", "age":39}").execute(); // Find a document DocResult docs = myColl.find("name like :name AND age < :age") .bind("name", "S%").bind("age", 20).execute(); // Print document DbDoc doc = docs.fetchOne(); System.out.println(doc); // Drop the collection myDB.dropCollection("test", "my_collection");
- 30. 30 New Shell
- 31. Starting using MySQL in few minutes 31
- 32. Quickly add a document 32
- 33. Find that document 33
- 35. Shell info 35
- 36. For this example, I will use the well known restaurants collection: We need to dump the data to a file and we will use the MySQL Shell with the Python interpreter to load the data. Migration from MongoDB to MySQL Document Store 36
- 37. Dump and load using MySQL Shell & Python This example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore $ mongo quiet eval 'DBQuery.shellBatchSize=30000; db.restaurants.find().shellPrint()' | perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json 37
- 38. Or use new bulk loader in 8.0.13 38
- 39. BSON Support Now, it supports the conversion of the following additional BSON types: ■ Date ■ Timestamp ■ NumberDecimal ■ NumberLong ■ NumberInt ■ Regular Expression ■ Binary 39 > util.importJson("/path_to_file/neighborhoods_mongo.json", {schema: "test", collection: "neighborhoods", convertBsonTypes: true});
- 40. 40
- 41. 41 Let’s query Too many records to show here … let’s limit it! restaurants.find().limit(1)
- 43. 43 Comparing Syntax: MongoDB vs MYSQL MongoDB: > db.restaurants.find({"cuisine": "French", "borough": { $not: /^Manhattan/} }, {"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2) MySQL: >restaurants.find(“cuisine=’French’ AND borough!=’Manhattan’”).fields([“name”,”cuisine”,”borough” ]).limit(2)
- 49. 49 MySQL Document Store Objects Summary
- 50. MySQL Document Store is Fully ACID Compliant 50
- 51. MySQL Document Store is Fully ACID Compliant 51
- 52. What about old SQL? The Hidden Part of the Iceberg 52
- 53. ★ Native data type (since 5.7.8) ★ JSON values are stored in MySQL tables using UTF8MB4 ★ Conversion from "native" SQL types to JSON values ★ JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...) ★ Generated/virtual columns ○ Indexing JSON data ○ Foreign Keys to JSON data ○ SQL Views to JSON data JSON datatype is behind the scene 53
- 54. How Does It Work?? 54
- 55. What does a collection look like on the server ? 55
- 56. Every document has a unique identifier called the document ID, which can be thought of as the equivalent of a table's primary key. The document ID value can be manually assigned when adding a document. If no value is assigned, a document ID is generated and assigned to the document automatically ! Use getDocumentId() or getDocumentIds() to get _ids(s) _id 56
- 57. Mapping to SQL Examples createCollection('mycollection') versus CREATE TABLE `test`.`mycoll` ( doc JSON, _id VARCHAR(32) GENERATED ALWAYS AS (doc->>'$._id') STORED PRIMARY KEY ) CHARSET utf8mb4; 57
- 58. Mapping to SQL Examples mycollection.add({‘test’: 1234}) versus INSERT INTO `test`.`mycoll` (doc) VALUES ( JSON_OBJECT( 'test',1234)); 58
- 59. More Mapping to SQL Examples mycollection.find("test > 100") Versus SELECT doc FROM `test`.`mycoll` WHERE (JSON_EXTRACT(doc,'$.test') >100); 59
- 60. 60 SQL and JSON Example
- 61. It's also possible to create indexes without using SQL syntax 61
- 62. SQL and JSON Example (2): validation 62
- 63. SQL and JSON Example (3): explain 63
- 64. SQL and JSON Example (3): explain 64
- 65. SQL and JSON Example (4): add index 65
- 66. SQL and JSON Example (4): add index 66
- 67. [ { "date": { "$date": 1416009600000 }, "grade": "Z", "score": 38 }, { "date": { "$date": 1398988800000 }, "grade": "A", "score": 10 }, { "date": { "$date": 1362182400000 }, "grade": "A", "score": 7 }, { "date": { "$date": 1328832000000 }, "grade": "A", "score": 13 } ] 67 Embedded Arrays of values can be messy to traverse.
- 68. SQL and JSON Example (5): arrays 68
- 69. $.grades[0] $.greades[1 to 2] $.grades[first] $.grades[last] $.grades[first to last - 1] 69 Arrays are now simple
- 70. NoSQL as SQL 70 JSON_TABLE turns your un- structured JSON data into a temporary structured table!
- 71. NoSQL as SQL 71 This temporary structured table can be treated like any other table -- LIMIT, WHERE, GROUP BY ...
- 73. Find the top 10 restaurants by grade for each cuisine 73 WITH cte1 AS (SELECT doc->>"$.name" AS name, doc->>"$.cuisine" AS cuisine, (SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT PATH "$.score")) AS r) AS avg_score FROM restaurants) SELECT *, RANK() OVER (PARTITION BY cuisine ORDER BY avg_score DESC) AS `rank` FROM cte1 ORDER BY `rank`, avg_score DESC LIMIT 10; This query uses a Common Table Expression (CTE) and a Windowing Function to rank the average scores of each restaurant, by each cuisine assembled in a JSON_TABLE
- 74. Conclusion: What Do I Gain? 74
- 75. This is the best of the two worlds in one product ! ● Data integrity ● ACID Compliant ● Transactions ● SQL ● Schemaless ● flexible data structure ● easy to start (CRUD) 75
- 76. Mutable Data!! Reduce Many to many joins Replace ‘stub’ tables Change on the fly, aggregate new data 76
- 77. Non JSON Data Transforms to JSON 77
- 78. And JSON to Relational SELECT country_name, IndyYear FROM countryinfo, JSON_TABLE(doc,"$" COLUMNS (country_name CHAR(60) PATH "$.Name",IndyYear INT PATH "$.IndepYear")) AS stuff WHERE IndyYear > 1992; +----------------+----------+ | country_name | IndyYear | +----------------+----------+ | Czech Republic | 1993 | | Eritrea | 1993 | | Palau | 1994 | | Slovakia | 1993 | 78
- 79. GeoJSON support too! mysql> SELECT ST_AsGeoJSON(ST_GeomFromText('POINT(11.11111 12.22222)'),2); +-------------------------------------------------------------+ | ST_AsGeoJSON(ST_GeomFromText('POINT(11.11111 12.22222)'),2) | +-------------------------------------------------------------+ | {"type": "Point", "coordinates": [11.11, 12.22]} | +-------------------------------------------------------------+ 79
- 80. New in MySQL 8.0 1. True Data Dictionary 2. Default UTF8MB4 3. Windowing Functions, CTEs, Lateral Derived Joins 4. InnoDB SKIPPED LOCK and NOWAIT 5. Instant Add Column 6. Histograms 7. Resource Groups 8. Better optimizer with new temporary table engine 9. True Descending Indexes 10.3D GIS 11.JSON Enhancements 80
- 81. Please buy my book! If you deal with the JSON Data Type or have an interest in the MySQL Document Store, this text is a great guide with many examples to help you understand the complexities and opportunities with a native JSON Data Type 81
Editor's Notes
- #81: What is new in MySQL 8.0