Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14-03-2017)
Download as PPTX, PDF6 likes4,923 views
The document provides an overview of NoSQL databases, particularly focusing on MongoDB, detailing its history, architecture, and operational characteristics. It contrasts traditional relational databases with NoSQL approaches, emphasizing the benefits of scalability, availability, and performance in modern application environments. The document also covers practical aspects such as installation, CRUD operations, indexing, and programming with MongoDB using various programming languages.
![RELATIONAL DATABASES
âą Based on relational model of data (E.F. Codd), a mathematical
foundation
âą Uses SQL for query, DML and DDL
âą Transactions are ACID (Atomicity, Consistency, Isolation, Durability)
âą All or nothing
âą Constraint Compliant
âą Individual experience [in a multi-session environment]
(aka concurrency)
âą Down does not hurt
6](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-6-320.jpg)
![THE RELATIONAL MODEL IN PRACTICE
âą Traditional Relational Data Model has severe impact on physical disk
performance
âą Transaction Log => Sequential Write (append to file)
âą Data Blocks require much more expensive Random Access disk writes
âą Indexes (B-Tree, Bitmap, âŠ) are used to speed up query (read) performance
âą and slow down transactions
âą Relational data does not [always] map naturally to data format required in
application (OO, JSON, XML)
âą Capability to join and construct ad-hoc queries across the entire data model is
powerful
âą Declarative integrity constraints allow for strict enforcement of data quality rules
âą âthe data may be non sensical, but at least it adheres to the rules, whatever the originâ
âą SQL has been embraced by huge numbers of tools & technologies and IT/Data
professionals](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-10-320.jpg)
![DATABASES RE-EVALUATED
âą Not all use cases require ACID (or can afford it)
âą Read only (product catalog for web shops)
âą Inserts only and no (inter-record) constraints
âą Big Data collected and âdumpedâ in Data Lake (Hadoop) for subsequent
processing
âą High performance demands
âą Not all data needs structured formats or structured querying and JOINs
âą Entire documents are stored and retrieved based on a single key
âą Sometimes â scalable availability is more important than Consistency â and
ACID is sacrificed
âą CAP-theorem states: Consistency [across nodes], Availability and Partition
tolerance can not all three be satisfied
11](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-11-320.jpg)
![ARCHITECTUUR, INSTALLATIE,
CONFIGURATIE, OPSLAG, SHELL
âą Mongo draait op windows en linux.
âą Windows install met msi file
âą linux install met tgz file.
âą Users en directories vrij te kiezen
âą Mongod --dbpath db1-- port 27001
âą Default path=datadb, default port=27017
âą Mongo [-- port hostname:27001] [--shell some_json_file]
âą Dbs, collections
25](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-25-320.jpg)
![RETRIEVE DATA (1)
Connect,
create,
update
var MongoClient = require('mongodb').MongoClient;
var mongodbHost = '127.0.0.1';
var mongodbPort = '27017';
var mongodbDatabase = 'world';
var url = 'mongodb://'+mongodbHost+':'+mongodbPort + '/' + mongodbDatabase;
MongoClient.connect(url, function(err, db) {
console.log("Connected correctly to server.");
db.collection('countries').find({},{"sort": [["area",-1]]})
.limit(20).toArray(function(err, results){
console.log("Name of Country Four " +results[3].name+ " and size: " +results[3].area);
// using cursors to retrieve data sets in a controlled fashion
// note: cursor implements NodeJS Stream - results can be piped
var cursor = db.collection('countries').find({"continent":"Asia"}, {"sort": "name"});
cursor.count(function(err, count){
console.log("Country count in Asia: "+count);
});
cursor.each(function(err, country){
console.log(country.name);
})//each cursor](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-40-320.jpg)
![RETRIEVE DATA (2)
Connect,
create,
update
var aggquery = [ {$sort: {area : -1}}
, {$group:{ _id: '$continent'
, largestCountry : {$first: "$name"}
}
}
];
var aggcursor = db.collection('countries').aggregate(aggquery);
aggcursor.each(function(err, result){
if (err) {
console.log(err);
} else if (result)
console.log(JSON.stringify(result));
}) //each aggcursor
var ccursor = db.collection('countries').find({});
// the cursor returned from find and aggregate implements a NodeJS (readable) Stream
ccursor.on('data', function(doc) {
console.log(doc);
});
ccursor.once('end', function() {
console.log(âOut of countries. Time to move on");
});](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-41-320.jpg)
![RETRIEVE DOCUMENTS â ECMA 6
Connect,
create,
update
co(function*() {
// Use connect method to connect to the Server
var db = yield MongoClient.connect(url);
console.log("Connected correctly to server");
// find top 20 countries by size
var results = yield db.collection('countries').find({}
,{"sort": [["area",-1]]}).limit(20).toArray();
console.log("Country One " +JSON.stringify(results[0]));
console.log("Name of Country Four " +results[3].name
+ " and size: " +results[3].area);
// use cursor to get the country count
var cursor = db.collection('countries').find({"continent":"Asia"}
, {"sort": "name"});
var count = yield cursor.count();
console.log("Country count in Asia: "+count);
while (yield cursor.hasNext()){
var country = yield cursor.next();
console.log(country.name);
}](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-43-320.jpg)
![RETRIEVE DOCUMENTS (2) â ECMA 6
Connect,
create,
update
// the largest country per continent
var aggquery = [ {$sort: {area : -1}}
, {$group:{ _id: '$continent'
, largestCountry : {$first: "$name"}
}
}
];
var aggcursor = db.collection('countries').aggregate(aggquery);
while (yield aggcursor.hasNext()){
var result = yield aggcursor.next();
console.log(JSON.stringify(result));
}
// Close the connection
db.close();
}).catch(function(err) {
console.log(err.stack);
});//co](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-44-320.jpg)
![CREATE DOCS IN MONGODB â ES6
Connect,
create,
update
// define two documents, any structure you like
var doc = {
"continent" : "Europe",
"children" : [ {"name":"Belgium"}, {"name":"Luxemburg"}],
"someVariable" : 19123, "andmore" : "2716NK" };
var doc2 = {
"continent" : "Asia", "name" : "Johnny",
"nrs" : [ {"name":"China"}, {"name":"India"}, {"name":"Buthan"}],
"tree": {"branch": {"twig":{"leaf": 1}}}
};
var nameOfCollection = "myDocs"
var result = yield db.collection(nameOfCollection).insertMany([doc,doc2]);
console.log(">> "+result.insertedCount
+" documents created into collection "+nameOfCollection);
...](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-45-320.jpg)
![CREATE & UPDATE DOCS â ES6
Connect,
create,
update
// define two documents, any structure you like
var doc = {...}, doc2 = {...} , nameOfCollection = "myDocsâ;
var result = yield db.collection(nameOfCollection).insertMany([doc,doc2]);
console.log(">> "+result.insertedCount +" documents created);
var cursor = db.collection(nameOfCollection).find();
while (yield cursor.hasNext()){
var doc = yield cursor.next();
console.log("Document: " +JSON.stringify(doc));
}// while cursor
result = yield db.collection(nameOfCollection).updateOne(
{"tree.branch.twig.leaf":1}, {$set: {name: "Hank", city:"Nieuwegeinâ
, "location.province":"Utrechtâ
, "location.country":"The Netherlandsâ
, "tree.stem":5}});
console.log(">> Updated "+result.modifiedCount+" document(s)");
cursor = db.collection(nameOfCollection).find();
while (yield cursor.hasNext()){
var doc = yield cursor.next();
console.log("Document: " +JSON.stringify(doc));
}// while cursor](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-46-320.jpg)
![CREATE, UPDATE AND DELETE â ES6
Connect,
create,
update
// define two documents, any structure you like
var doc = {...}, doc2 = {...} , nameOfCollection = "myDocsâ;
var result = yield db.collection(nameOfCollection).insertMany([doc,doc2]);
console.log(">> "+result.insertedCount +" documents created);
var cursor = db.collection(nameOfCollection).find();
while (yield cursor.hasNext()){
var doc = yield cursor.next();
console.log("Document: " +JSON.stringify(doc));
}// while cursor
result = yield db.collection(nameOfCollection).updateOne(...);
...
result = yield db.collection(nameOfCollection).deleteMany({});
console.log(">> Deleted "+ result.deletedCount+" documents ");
// execute command to drop the collection
yield db.command({drop:nameOfCollection});
console.log(">> Dropped collection "+nameOfCollection);
// Close the connection
db.close();
console.log("Connection to database is closed.");](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-47-320.jpg)
![CONNECTING TO SERVER
package nl.amis.mongodb.countries;
import java.net.UnknownHostException;
import com.mongodb.MongoClient;
import com.mongodb.client.MongoDatabase;
public class AppConnect {
public static void main(String[] args) throws UnknownHostException {
String mongodbHost = "127.0.0.1";
Integer mongodbPort = 27017;
String mongodbDatabase = "world";
MongoClient mongoClient = new MongoClient(mongodbHost, mongodbPort);
try {
System.out.println("Connected to server; now hook into database " + mongodbDatabase);
MongoDatabase db = mongoClient.getDatabase(mongodbDatabase);
System.out.println("Connected to database " + mongodbDatabase + " successfully");
} finally {
mongoClient.close();
System.out.println("Closed mongodb connection");
}
}
}
Connect,
create,
update](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-49-320.jpg)
![RETRIEVE DATA
...
import org.bson.Document;
import com.mongodb.client.MongoCollection;
public class AppConnect {
public static void main(String[] args) throws UnknownHostException {
... Connect to world database
// get all documents from the countries collection
MongoCollection<Document> countries = db.getCollection("countries");
System.out.println("Got collection countries successfully");
System.out.println("Number of countries: " + countries.count());
System.out.println("All countries - name and continent");
for (Document doc : countries.find()) {
// System.out.println(doc.toJson());
System.out.println(doc.get("name") + " (" + doc.get("continent") + ")");
}//for
Connect,
create,
update](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-50-320.jpg)
![RETRIEVE DATA (2)
import com.mongodb.client.model.Sorts;
import com.mongodb.client.model.Filters;
import com.mongodb.Block;
public class AppConnect {
public static void main(String[] args) throws UnknownHostException {
... Connect to world database
System.out.println( "Retrieve all countries in Asia order alphabetically
by name and display full country document");
countries.find(Filters.eq("continent", "Asia"))
.sort(Sorts.ascending("name"))
.forEach(new Block<Document>() {
@Override
public void apply(final Document document) {
System.out.println(document.toJson());
}
});
Connect,
create,
update](https://image.slidesharecdn.com/nosql-and-mongodb-sig-14march2017-170315090521/85/Introducing-NoSQL-and-MongoDB-to-complement-Relational-Databases-AMIS-SIG-14-03-2017-51-320.jpg)
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14-03-2017)
- 1. INTRODUCING NOSQL AND ONGODB Pom Bleeksma & Lucas Jellema 14 March 2017, Nieuwegein M
- 2. AGENDA INTRODUCTION WHATâS NOT SQL ABOUT NOSQL â HISTORY, BACKGROUND, USE CASES OVERVIEW OF NOSQL DATABASES INTRODUCING MONGODB GETTING STARTED WITH MONGODB â INSTALLATION, CONFIGURATION, ADMINISTRATION DEVELOPING WITH MONGODB â JAVA & NODE.JS HANDSON WORKSHOP
- 3. TRADITIONAL APPROACH: ALL DATA IN ENTERPRISE RELATIONAL DATABASE
- 4. TRADITIONAL APPROACH: WITHOUT MUCH REGARD FOR âą What is it used for? âą By whom, where, in what way, using which tools âą Ratio between writes and reads âą Ad hoc access? âą What format is it in/should it be in? âą What confidentiality & integrity is required? âą How much of it? âą How long relevant (hot vs cold vs dead)? âą How fine grained and how accurate? âą How much change?
- 5. DATABASE: SINGLE POINT OF ⊠SCALABILITY | AVAILABILITY | CONSISTENCY âą Single Point of ⊠⹠What goes where âŠ
- 6. RELATIONAL DATABASES âą Based on relational model of data (E.F. Codd), a mathematical foundation âą Uses SQL for query, DML and DDL âą Transactions are ACID (Atomicity, Consistency, Isolation, Durability) âą All or nothing âą Constraint Compliant âą Individual experience [in a multi-session environment] (aka concurrency) âą Down does not hurt 6
- 7. THE HOLY GRAIL OF NORMALIZATION âą Normalize to prevent data redundancy, discrepancies (split brain) and storage waste âą Pragmatic approaches - recognizing the fact that some data is read far more frequently than that it is created and modified âą Materialized View âą Virtual Column âą Index âą Sharding âą Replication âą Data Guard âą Data Warehouse & Data Marts âą Client Side Caching âą In Memory Database
- 8. ACID COMES AT A COST âą Transaction results have to be persisted in order to guarantee D âą Concurrency requires some degree of locking (and multi-versioning) in order to have I âą Constraint compliance (unique key, foreign key) means all data hangs together (as do all transactions) in order to have C 8
- 9. ACID COMES AT A COST âą ACID means: âą All data closely together (and sometimes far from applications and users) âą All transactions centrally organized âą All data persisted as part of the transaction âą ACID means: âą Hard to scale out (horizontally) âą Oracle offers scale up with Real Application Clusters 9
- 10. THE RELATIONAL MODEL IN PRACTICE âą Traditional Relational Data Model has severe impact on physical disk performance âą Transaction Log => Sequential Write (append to file) âą Data Blocks require much more expensive Random Access disk writes âą Indexes (B-Tree, Bitmap, âŠ) are used to speed up query (read) performance âą and slow down transactions âą Relational data does not [always] map naturally to data format required in application (OO, JSON, XML) âą Capability to join and construct ad-hoc queries across the entire data model is powerful âą Declarative integrity constraints allow for strict enforcement of data quality rules âą âthe data may be non sensical, but at least it adheres to the rules, whatever the originâ âą SQL has been embraced by huge numbers of tools & technologies and IT/Data professionals
- 11. DATABASES RE-EVALUATED âą Not all use cases require ACID (or can afford it) âą Read only (product catalog for web shops) âą Inserts only and no (inter-record) constraints âą Big Data collected and âdumpedâ in Data Lake (Hadoop) for subsequent processing âą High performance demands âą Not all data needs structured formats or structured querying and JOINs âą Entire documents are stored and retrieved based on a single key âą Sometimes â scalable availability is more important than Consistency â and ACID is sacrificed âą CAP-theorem states: Consistency [across nodes], Availability and Partition tolerance can not all three be satisfied 11
- 12. USE CASE: WEBSHOP âą Webshop â 1M visitors per day âą Product catalog consists of millions of records âą The web shop presents: product description, images, reviews, pricing details, related offerings, stock status âą Products are added and updated and removed every day âą Although most products do not change very frequently âą Some vendors do bulk manipulation of product details Products Product updates Webshop visits - searches - product details - orders
- 13. USE CASE: WEBSHOP TECHNOLOGY AND ARCHITECTURE CONSIDERATIONS Products Product updates Webshop visits - searches - product details - orders firewall Data manipulation Data Quality (enforcement) <10K transactions Batch jobs next to online Speed is nice Read only On line Speed is crucial XHTML & JSON > 1M visits
- 14. Products Products USE CASE: WEBSHOP TECHNOLOGY AND ARCHITECTURE CONSIDERATIONS Products Webshop visits - searches - product details - orders firewall Data manipulation Data Quality (enforcement) <10K transactions Batch jobs next to online Speed is nice Read only On line Speed is crucial XHTML & JSON > 1M visits DMZ Read only JSON documents Images Text Search Scale Horizontally Products Nightly generation Product updates
- 15. A FEW OBSERVATIONS âą Redundancy âą for performance, scalability, availability âą Commodity hardware for scale out architecture âą Data (documents) prepared for specific application usage (super MV) âą Format âą Search paths âą Time lag in data modifications is accepted âą Perhaps even desirable to offer a consistent view to webshop users âą Transactions in one engine âą ACID still in place âą Transactions do not compete with reads âą vying for the same physical resources
- 16. NOSQL AND BASE ⹠NoSQL arose because of performance and scalability challenges with Web Scale operations ⹠NoSQL is a label for a wide variety of databases that lack some aspect of a true relational database ⹠ACID-ness, SQL, relational model, constraints ⹠The label has been used since 2009 ⹠Perhaps NoREL would be more appropriate ⹠Some well known NoSQL products are ⹠Cassandra, MongoDB, Redis, CouchDB, ⊠⹠BASE as alternative to ACID: ⹠basically available, soft state, eventually consistent (after a short duration) 16
- 17. TYPICAL FOR NOSQL âą Focus on speed, availability and scalability âą Horizontal scale out â distributed with load balancing and fail-over âą No (predefined) Data Structure âą Integrity primarily protected by application logic âą Open Source (most offerings are, not all: MarkLogic) âą Close(r) attention for how the data is used âą Application oriented data format and search paths and specialized database per application (microservice, capability) âą Similar to the switch from SOA to API/Microservice âą Reads (far) more relevant than writes âą Data redundancy & denormalization âą No data access through SQL
- 18. NOSQL MEANS: NO DATA ACCESS THROUGH SQL âą However âą Data Professionals and Developers speak SQL âą Reporting, Dashboarding, ETL, BI tools speak SQL âą There is no common query language across NoSQL products
- 19. NO DATA ACCESS THROUGH SQL âą However âą Data Professionals and Developers speak SQL âą Reporting, Dashboarding, ETL, BI tools speak SQL âą There is no common query language across NoSQL products âą Attempts from many vendors to create drivers that translate SQL statements into NoSQL commands for the specific target database âą To protect existing investments in SQL â skills, tools, applications, reports, ..
- 20. TYPES OF NOSQL DATABASES âą Key-Value (from in-memory cache to hardware appliance â Memcached, Hazelcast, Coherence, Redis, Oracle NoSQL/Berkeley DB) âą Wide Column Store (Cassandra, Google BigTable, HBase) âą Document (JSON, XML, YAML and binary forms â MongoDB, CouchDB, Couchbase, MarkLogic) âą Graph (networks of data, triplets, RDF stores â Neo4J) âą Miscellaneous: object, tabular, tuple, elastic search index, Hadoop (hdfs) style, Kafka Topic
- 21. (LEADING) NOSQL DATABASE PRODUCTS ⹠MongoDB is (one of) the most popular (by any measure) ⹠Cloud (only): ⹠Google BigTable, ⹠AWS Dynamo ⹠Cache (in memory) ⹠ZooKeeper, Redis, Memcached, ⊠⹠Hadoop/HDFS ⹠Oracle NoSQL (fka Berkeley DB)
- 23. HISTORY OF MONGODB âą 10gen â startup from 2007 in New York City âą Developed database as component in Platform as a Service product âą Initial release of MongoDB: 2009 âą 10gen changed its name to MongoDB Inc. in 2013 âą Offers enterprise support, MongoDB Atlas cloud service, MongoDB Compass, Connectors âą Current stable release: 3.4.2 (3.5.2 is in preview) âą Open Source - GNU Affero General Public License and the Apache License. âą MongoDB == Humongous Database âą Evolution of MongoDB is still pretty fast âą Some crucial enterprise database capabilities were added fairly recently or are still lacking âą MongoDB is number one NoSQL database in terms of popularity
- 24. CORE ARCHITECTURE âą MongoDB stores JSON documents in a binary format (BSON) âą Documents are stored in collections (similar to row or records in tables) âą Interaction is through JavaScript âą taking the place of SQL for DML and query âą Written in C++ âą Has the V8 JavaScript engine included âą Runs on Mac OS X, Windows, Solaris, and most flavors of Linux âą Source code is on GitHub âą Has GeoSpatial and Tekst indexes and search capabilities âą MongoDB, Inc. officially supports drivers for C, C++, C#, Erlang, Java, Node.js, JavaScript, Perl, PHP, Python, Scala, and Ruby
- 25. ARCHITECTUUR, INSTALLATIE, CONFIGURATIE, OPSLAG, SHELL âą Mongo draait op windows en linux. âą Windows install met msi file âą linux install met tgz file. âą Users en directories vrij te kiezen âą Mongod --dbpath db1-- port 27001 âą Default path=datadb, default port=27017 âą Mongo [-- port hostname:27001] [--shell some_json_file] âą Dbs, collections 25
- 26. IMPORT EXPORT âą Mongoimport âą Json âą Csv, tsv, maar hebben een descriptor file nodig âą mongoexport -h -d -c -o -q âą Leesbare output, minder geschikt voor backups
- 27. CRUD âą show dbs âą use db âą db.<collection>.find() âą db.<collection>.find().pretty() âą select: db.find({key:value,key:value}) âą present: db.<collection>.find({key:value},{key:1,key:1,_id:0}) âą explain: db.find({key:value,key:value}).explain("executionStats") âą db.companies.insert({Company:"AMIS",Event:"MongoDB Workshop"}) âą db.companies.update({Company:"AMIS"},{$set: {Location:"Nieuwegein"}}) âą Geen commit! Ofwel, autocommit.
- 28. CRUD âą db.companies.remove({"Company" : "AMIS"}) âą $unset -> Om een veld uit een document te verwijderen. âą $inc -> om een numeriek veld te verhogen met een bepaalde waarde. Als het veld niet bestaat wordt het toegevoegd (met de waarde van de verhoging) âą $push -> Om een waarde aan een array toe te voegen.
- 29. INDEXING, EXPLAIN âą _Id altijd geĂŻndexeerd en unique âą Indexen helpen bij zoeken âą Indexen mogen compound zijn, dwz op meerdere keys gecombineerd, bv index op woonplaats+leeftijd âą db.user.find().sort({ woonplaats:1, leeftijd: -1 }) De -1 betekent aflopend geĂŻndexeerd. âą Resultaten rechtstreeks uit index indien mogelijk (fat indexes in SQL) âą db.user.find({woonplaats:âZoetermeerâ}).explain(âexecutionStatsâ) geeft een goed overzicht van het gevolgde plan
- 30. REPLICA SETS âą Replica sets zijn gedupliceerde collections âą Beveilliging tegen uitval âą De members besluiten onderling welk member primary is. Alleen daarop kan DML plaatsvinden âą Altijd oneven aantal members, eventueel mbv een arbiter âą Secondary members kunnen wel voor select gebruikt worden, als je eerst slaveOk() ingeeft âą Mogen op dezelfde host draaien, mogen ook op verschillende hosts draaien. âą writeConcern: Je kan expliciet aangeven op hoeveel members je DML uitgevoerd moet zijn voor je je prompt terug krijgt.
- 31. SHARDING âą Shard = splinter âą Range based partitioning âą Altijd index nodig op de shard key. Hoeft niet unique te zijn. âą Iedere shard is een proces. Bij voorkeur ieder proces zijn eigen host. âą Scale out proces âą Metadata staat in aparte config db. Typically 3 stuks. âą Clients verbinden met een mongos server. Dat is een instance zonder database die uit de config haalt waar de data zich bevindt. âą Iedere shard kan bestaan uit replica sets.
- 32. SHARDING âą Data verdeelt zichzelf over chunks. âą Chunks kunnen splitten wanneer ze te groot worden (typically 100MB maar is aan te passen naar wens) âą Chunks kunnen migraten naar een andere chard. Mongo streeft ernaar elke shard evenveel chunks te laten hebben. âą Tijdens migration kan DML gewoon doorgaan.
- 33. BACKUP & RESTORE ⹠Backup met mongodump ⹠Creëert een dump directory met daaronder een directory per database en plaatst daarin 1 of meerdere .bson files, een binair formaat. ⹠Kan vanaf ieder member van een replica set. Bij voorkeur eentje die het niet druk heeft. ⹠Mongo kent geen transactions, dus backup is niet consistent. Het is maar afwachten wat erin staat als er veel DML plaatsvindt. ⹠De tegenhanger heet mongorestore. ⹠Restore van bestaande database lukt niet: duplicate key op _id
- 34. MONGODB COMPASS (AKA NOSQL DEVELOPER) ⹠The GUI for MongoDB ⹠Visually explore data ⹠Run ad hoc queries in seconds. ⹠Interact with your data with full CRUD functionality ⹠View and optimize your query performance. ⹠Separately installed product ⹠On Windows, OS X, Linux, ⊠⹠Can run against local and remote MongoDB servers
- 35. MONGODB COMPASS
- 36. PROGRAMMING AGAINST MONGODB âą MongoDB officially supports drivers for C, C++, C#, Erlang, Java, Node.js, JavaScript, Perl, PHP, Python, Scala, and Ruby âą Note: there is also a Connector for Hadoop âą Several REST APIs for MongoDB are available from the community âą RESTHeart (Java), Eve (Python),, Crest (Node), AMID, Kule, DreamFactory Connect, create, update Connect, create, update
- 37. MEAN STACK âą Modelled after LAMP âą End-to-End JavaScript/JSON âą Term coined in 2013 (by MongoDB)
- 38. INTERACTING WITH MONGODB FROM NODE.JS âą MongoDB Node.js Driver 2.0 âą Support for ECMAScript 6.0 â Promises for asynch âą http://mongodb.github.io/node-mongodb-native/2.0/ âą Using mongodb in a NodeJS application âą npm install mongodb âsave âą var MongoClient = require('mongodb').MongoClient; Connect, create, update
- 39. CONNECTING TO SERVER Connect, create, update var MongoClient = require('mongodb').MongoClient; var mongodbHost = '127.0.0.1'; var mongodbPort = '27017'; var mongodbDatabase = 'world'; var url = 'mongodb://'+mongodbHost+':'+mongodbPort + '/' + mongodbDatabase; MongoClient.connect(url, function(err, db) { console.log("Connected correctly to server."); // DO YOUR THING WITH MONGODB db.close(); console.log("Connection to database is closed."); }) //connect()
- 40. RETRIEVE DATA (1) Connect, create, update var MongoClient = require('mongodb').MongoClient; var mongodbHost = '127.0.0.1'; var mongodbPort = '27017'; var mongodbDatabase = 'world'; var url = 'mongodb://'+mongodbHost+':'+mongodbPort + '/' + mongodbDatabase; MongoClient.connect(url, function(err, db) { console.log("Connected correctly to server."); db.collection('countries').find({},{"sort": [["area",-1]]}) .limit(20).toArray(function(err, results){ console.log("Name of Country Four " +results[3].name+ " and size: " +results[3].area); // using cursors to retrieve data sets in a controlled fashion // note: cursor implements NodeJS Stream - results can be piped var cursor = db.collection('countries').find({"continent":"Asia"}, {"sort": "name"}); cursor.count(function(err, count){ console.log("Country count in Asia: "+count); }); cursor.each(function(err, country){ console.log(country.name); })//each cursor
- 41. RETRIEVE DATA (2) Connect, create, update var aggquery = [ {$sort: {area : -1}} , {$group:{ _id: '$continent' , largestCountry : {$first: "$name"} } } ]; var aggcursor = db.collection('countries').aggregate(aggquery); aggcursor.each(function(err, result){ if (err) { console.log(err); } else if (result) console.log(JSON.stringify(result)); }) //each aggcursor var ccursor = db.collection('countries').find({}); // the cursor returned from find and aggregate implements a NodeJS (readable) Stream ccursor.on('data', function(doc) { console.log(doc); }); ccursor.once('end', function() { console.log(âOut of countries. Time to move on"); });
- 42. CONNECTING TO SERVER â ECMA 6 Connect, create, update var MongoClient = require('mongodb').MongoClient; var assert = require('assert'), co = require('co'); var mongodbHost = '127.0.0.1'; var mongodbPort = '27017'; var mongodbDatabase = 'world'; var url = 'mongodb://'+mongodbHost+':'+mongodbPort + '/' + mongodbDatabase; co(function*() { // Use connect method to connect to the Server var db = yield MongoClient.connect(url); console.log("Connected correctly to server"); // DO YOUR THING WITH MONGODB // Close the connection db.close(); }).catch(function(err) { console.log(err.stack); });
- 43. RETRIEVE DOCUMENTS â ECMA 6 Connect, create, update co(function*() { // Use connect method to connect to the Server var db = yield MongoClient.connect(url); console.log("Connected correctly to server"); // find top 20 countries by size var results = yield db.collection('countries').find({} ,{"sort": [["area",-1]]}).limit(20).toArray(); console.log("Country One " +JSON.stringify(results[0])); console.log("Name of Country Four " +results[3].name + " and size: " +results[3].area); // use cursor to get the country count var cursor = db.collection('countries').find({"continent":"Asia"} , {"sort": "name"}); var count = yield cursor.count(); console.log("Country count in Asia: "+count); while (yield cursor.hasNext()){ var country = yield cursor.next(); console.log(country.name); }
- 44. RETRIEVE DOCUMENTS (2) â ECMA 6 Connect, create, update // the largest country per continent var aggquery = [ {$sort: {area : -1}} , {$group:{ _id: '$continent' , largestCountry : {$first: "$name"} } } ]; var aggcursor = db.collection('countries').aggregate(aggquery); while (yield aggcursor.hasNext()){ var result = yield aggcursor.next(); console.log(JSON.stringify(result)); } // Close the connection db.close(); }).catch(function(err) { console.log(err.stack); });//co
- 45. CREATE DOCS IN MONGODB â ES6 Connect, create, update // define two documents, any structure you like var doc = { "continent" : "Europe", "children" : [ {"name":"Belgium"}, {"name":"Luxemburg"}], "someVariable" : 19123, "andmore" : "2716NK" }; var doc2 = { "continent" : "Asia", "name" : "Johnny", "nrs" : [ {"name":"China"}, {"name":"India"}, {"name":"Buthan"}], "tree": {"branch": {"twig":{"leaf": 1}}} }; var nameOfCollection = "myDocs" var result = yield db.collection(nameOfCollection).insertMany([doc,doc2]); console.log(">> "+result.insertedCount +" documents created into collection "+nameOfCollection); ...
- 46. CREATE & UPDATE DOCS â ES6 Connect, create, update // define two documents, any structure you like var doc = {...}, doc2 = {...} , nameOfCollection = "myDocsâ; var result = yield db.collection(nameOfCollection).insertMany([doc,doc2]); console.log(">> "+result.insertedCount +" documents created); var cursor = db.collection(nameOfCollection).find(); while (yield cursor.hasNext()){ var doc = yield cursor.next(); console.log("Document: " +JSON.stringify(doc)); }// while cursor result = yield db.collection(nameOfCollection).updateOne( {"tree.branch.twig.leaf":1}, {$set: {name: "Hank", city:"Nieuwegeinâ , "location.province":"Utrechtâ , "location.country":"The Netherlandsâ , "tree.stem":5}}); console.log(">> Updated "+result.modifiedCount+" document(s)"); cursor = db.collection(nameOfCollection).find(); while (yield cursor.hasNext()){ var doc = yield cursor.next(); console.log("Document: " +JSON.stringify(doc)); }// while cursor
- 47. CREATE, UPDATE AND DELETE â ES6 Connect, create, update // define two documents, any structure you like var doc = {...}, doc2 = {...} , nameOfCollection = "myDocsâ; var result = yield db.collection(nameOfCollection).insertMany([doc,doc2]); console.log(">> "+result.insertedCount +" documents created); var cursor = db.collection(nameOfCollection).find(); while (yield cursor.hasNext()){ var doc = yield cursor.next(); console.log("Document: " +JSON.stringify(doc)); }// while cursor result = yield db.collection(nameOfCollection).updateOne(...); ... result = yield db.collection(nameOfCollection).deleteMany({}); console.log(">> Deleted "+ result.deletedCount+" documents "); // execute command to drop the collection yield db.command({drop:nameOfCollection}); console.log(">> Dropped collection "+nameOfCollection); // Close the connection db.close(); console.log("Connection to database is closed.");
- 48. INTERACTING WITH MONGODB FROM JAVA âą MongoDB Java Driver 3.x âą Supports Synchronous and Asynchronous operations âą https://docs.mongodb.com/ecosystem/drivers/java/ âą Using MongoDB Driver in a Java Application âą Add dependency in Maven pom.xml file âą Library JAR file mongo-java-driver-3.4.0.jar Connect, create, update
- 49. CONNECTING TO SERVER package nl.amis.mongodb.countries; import java.net.UnknownHostException; import com.mongodb.MongoClient; import com.mongodb.client.MongoDatabase; public class AppConnect { public static void main(String[] args) throws UnknownHostException { String mongodbHost = "127.0.0.1"; Integer mongodbPort = 27017; String mongodbDatabase = "world"; MongoClient mongoClient = new MongoClient(mongodbHost, mongodbPort); try { System.out.println("Connected to server; now hook into database " + mongodbDatabase); MongoDatabase db = mongoClient.getDatabase(mongodbDatabase); System.out.println("Connected to database " + mongodbDatabase + " successfully"); } finally { mongoClient.close(); System.out.println("Closed mongodb connection"); } } } Connect, create, update
- 50. RETRIEVE DATA ... import org.bson.Document; import com.mongodb.client.MongoCollection; public class AppConnect { public static void main(String[] args) throws UnknownHostException { ... Connect to world database // get all documents from the countries collection MongoCollection<Document> countries = db.getCollection("countries"); System.out.println("Got collection countries successfully"); System.out.println("Number of countries: " + countries.count()); System.out.println("All countries - name and continent"); for (Document doc : countries.find()) { // System.out.println(doc.toJson()); System.out.println(doc.get("name") + " (" + doc.get("continent") + ")"); }//for Connect, create, update
- 51. RETRIEVE DATA (2) import com.mongodb.client.model.Sorts; import com.mongodb.client.model.Filters; import com.mongodb.Block; public class AppConnect { public static void main(String[] args) throws UnknownHostException { ... Connect to world database System.out.println( "Retrieve all countries in Asia order alphabetically by name and display full country document"); countries.find(Filters.eq("continent", "Asia")) .sort(Sorts.ascending("name")) .forEach(new Block<Document>() { @Override public void apply(final Document document) { System.out.println(document.toJson()); } }); Connect, create, update
- 52. RETRIEVE DATA - AGGREGATION ... Block<Document> printBlock = new Block<Document>() { @Override public void apply(final Document document) { System.out.println("Country: " + document.toJson()); } }; ... System.out.println("List largest country in each continent"); countries.aggregate( Arrays.asList( Aggregates.sort(Sorts.descending("area")), Aggregates.group("$continent", Accumulators.first("Largest Country", "$name")) )) .forEach(printBlock); Connect, create, update
- 53. RETRIEVE DATA â AGGREGATION (2) System.out.println("List name in uppercase,sorted by size, delta between birthdate and deathrate, " +" population density for countries in Oceania"); BsonArray subArgs=new BsonArray(); subArgs.add(new BsonString("$birthrate")); subArgs.add(new BsonString("$deathrate")); BsonArray divArgs=new BsonArray(); divArgs.add(new BsonString("$population")); divArgs.add(new BsonString("$area")); countries.aggregate (Arrays.asList( Aggregates.match(Filters.eq("continent", "Oceania")), Aggregates.sort(Sorts.descending("area")), Aggregates.project( Projections.fields( Projections.excludeId(), Projections.include("continent"), Projections.computed("name",Projections.computed("$toUpper", "$name")), Projections.computed("populationGrowthRate",Projections.computed("$subtract", subArgs)), Projections.computed("populationDensity", Projections.computed("$trunc", Projections.computed("$divide", divArgs))) )) )) .forEach(printBlock); Connect, create, update
- 54. JAVA: CREATE DOCUMENTS IN MONGODB ... String nameOfCollection = "myDocs"; MongoCollection<Document> coll = db.getCollection(nameOfCollection); Document doc1 = new Document("continent", "Europe") .append("nrs", Arrays.asList(new Document("name", "Belgium"), new Document("name", "Luxemburg"))) .append("someVariable", 19123) .append("andmore", "kl;jdsfakhfsdahjfsdasjbsdahjbsdahgvsahjkZl;po"); Document doc2 = new Document("continent", "Asia").append("name", "Johnny") .append("nrs", Arrays.asList(new Document("name", "China"), new Document("name", "India"), new Document("name", "Buthan"))) .append("tree", new Document("branch", new Document("twig", new Document("leaf", 1)))); List<Document> documents = new ArrayList<Document>(); documents.add(doc1); documents.add(doc2); coll.insertMany(documents); System.out.println(">> Documents created into collection"); coll.find().forEach(printBlock); Connect, create, update
- 55. JAVA: UPDATE AND DELETE DATA ... coll.updateOne( Filters.eq("tree.branch.twig.leaf", 1), Updates.combine( Updates.set("name", "Hank"), Updates.set("city", "Nieuwegein"), Updates.set("location.province", "Utrecht"), Updates.set("location.country", "The Netherlands"), Updates.set("tree.stemp", 5), Updates.currentDate("lastModified") ) ); System.out.println(">> Updated one document in collection " + nameOfCollection); coll.find().forEach(printBlock); DeleteResult dr = coll.deleteMany(Filters.exists("_id")); System.out.println(">> Deleted " + dr.getDeletedCount() + " documents from collection " + nameOfCollection); db.runCommand(new Document("drop", nameOfCollection)); System.out.println(">> Dropped collection " + nameOfCollection); Connect, create, update
- 56. INTERACTING WITH MONGODB FROM JAVA â JSON ïł OO MAPPING âą Libraries for mapping between BSON/JSON documents and Java Objects âą Morphia - http://mongodb.github.io/morphia/1.3/getting-started/quick-tour/ âą Jongo â Query in Java as in the MongoDB shell - http://jongo.org/ Connect, create, update
- 57. HANDS ON WORKSHOP âą Install MongoDB âą Create Database, Create Collections and Documents âą Retrieve Data âą Import Database âą More configuration and administration âą Development: interacting with MongoDB from âą Compass âą Java âą Node.js
- 58. âą Resources: https://github.com/lucasjellema/sig-nosql-mongodb âą Blog: technology.amis.nl âą Email: [email protected] , [email protected] âą : @lucasjellema âą : lucas-jellema âą : www.amis.nl, [email protected] +31 306016000 Edisonbaan 15, Nieuwegein