MongoDB from Basics to Scale

© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for BillRun!© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for BillRun!
MongoDBFor BillRun!
Copyrights © Moshe Kaplan
moshe.kaplan@brightaqua.com

MongoDB
For BillRun!
Moshe Kaplan
Scale Hacker
http://top-performance.blogspot.com
http://blogs.microsoft.co.il/vprnd

© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for BillRun!
It’s all About
3
Scale

NOSQL. ANSWER A NEED
Introduction
4

5

The Consumer Revolution
6
http://topyaps.com/wp-content/uploads/2013/03/You-are-the-
product.-You-feeling-something.jpg

At the fraction of the cost…
7

8
http://lifehacker.com/5697167/if-youre-not-paying-for-it-
youre-the-product

Transportation
9

Moovit
10

The Medical Market Opportunities
11

MediSafe
12

13

Askem
14

Major Enablers:
Mobile, Cloud and IT Commoditization
15

The Prime Suspect
16

17
Assumptions…

Where did it Fail?
Get an Answer, Fast and Cheap

Where did it Fail?
I Just Want “Class Persistency
Storage” and Changing Schema on
Demand

Where did it Fail?
Be Always Available, Even w/ an Old
Answer

Where did it Fail?
Get Me Fast and Good Enough
Answer

Where did it Fail?
Data is Too Big, and Storage is $$$
But CPU and Network are Even More
http://www.powerbyte.com/Isilon.html

Software Providers
23

It is all great, but…
I Need to Meet Compliance
http://www.vision7.com/app_system/lib/image/content/PCI_compliance.jpg

I Need a Vendor

I Need Reporting
http://www.novell.com/communities/node/5851/get-ready-sentinel-61

I Need Transactions
http://www.novell.com/communities/node/5851/get-ready-sentinel-61

We Need Training for the Data Analysts
db.article.aggregate(
{ $group : {
_id : "$author",
docsPerAuthor : { $sum : 1 },
viewsPerAuthor : { $sum : "$pageViews" }
}}
);
< SUM(pageViews)
< SUM(1) = N
< GROUP BY author

NOSQL MARKET
Introduction
29

When Should I Choose NoSQL?
• Eventually Consistent
• Document Store
• Key Value
30
http://guyharrison.squarespace.com/blog/tag/nosq

Key Value Store
• insert
• get
• multiget
• remove
• truncate
31
<Key, Value>
http://wiki.apache.org/cassandra/API

Redis
• Very simple protocol (SMTP like)
• Amazing Performance (60Kqps ops on 1 CPU machine)
• Persistency to disk
• Very little security

Column Family Stores:
Key Value Store (with benefits)
• insert
• get
• multiget
• remove
• truncate
33
http://wiki.apache.org/cassandra/API

Cassandra
• Simple protocol
• Very Good Performance
• You have indexes (but limited)
• Data Model is a pain
• You need to design you data for queries:
“Table per Query”

Document Databases
var mydoc = {
_id: ObjectId("5099803df3f4948bd2f98391"),
name: { first: "Alan", last: "Turing" },
birth: new Date('Jun 23, 1912'),
death: new Date('Jun 07, 1954'),
contribs: [
"Turing machine",
"Turing test",
"Turingery"
],
views : NumberLong(1250000)
}
35

Database for Software Engineers
Class
Subclass
Document
Subdocument

37
MapReduce
http://blogs.microsoft.co.il/blogs/vprnd

HELLO. MY NAME IS MONGODB
Introduction
38

39
#5 Most Popular DB Engine
http://db-engines.com/en/ranking

Who is Using mongoDB?

Who is Behind mongoDB

Why MongoDB?
What? Why?
JSON End to End
No Schema “No DBA”, Just Serialize
Write 10K Inserts/sec on virtual machine
Read Similar to MySQL
HA 10 min to setup a cluster
Sharding Out of the Box
GeoData Great for that
No Schema None: no downtime to create new columns
Buzz Trend is with NoSQL
42

What mongoDB is Made of?
43
http://www.10gen.com/products/mongodb

Installation: Give Yourself 5min
• Add to /etc/yum.repos.d/10gen.repo
• [10gen]
• name=10gen Repository
• baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
• gpgcheck=0
• enabled=1
• yum –y install mongo-10gen mongo-10gen-server
• The Packages:
• mongo-10gen: tools
• mongo-10gen-server: mongod and mongos

The Ubuntu Way
• sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv
7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0
multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get -y update
sudo apt-get install -y mongodb-org

Installation w/ Authentication
• /etc/mongod.conf
• > mongo
• use admin
db.createUser(
{
user: "siteUserAdmin",
pwd: “Pss0rdxxx",
roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
} )

Mastering a New Query
Language

Connect to the Database
• Connect:
• > mongo
• Show current database:
• >> db
• Show Databases
• >> show databases;
• Show Collections
• >> show collections; or show tables;

Databases Manipulation: Create & Drop
• Change Database:
• >> use <database>
• Create Database
• Just switch and create an object…
• Delete Database
• > use mydb;
• > db.dropDatabase();

Collections Manipulation
• Create Collcation
>db.createCollection(collectionName)
• Delete Collection
> db.collectionName.drop()
Or just insert to it

SELECT: No SQL, just ORM…
• Select All
• db.things.find()
• WHERE
• db.posts.find({“comments.email” : ”b@c.com”})
• Pattern Matching
• db.posts.find( {“title” : /mongo/i} )
• Sort
• db.posts.find().sort({email : 1, date : -1});
• Limit
• db.posts.find().limit(3)

NoSQL and Data Modeling
What is the Difference

Same Terminology
• Database  Database
• Table  Collection
• Row  Document

A Blog Case Study in MySQL
http://www.slideshare.net/nateabele/building-apps-with-mongodb

as a SW Engineer would like it to be…

Migration from RDBMS to NoSQL
How to do that?

Data Migration
• Map the table structure
• Export the data and Import It
• Add Indexes
58
http://igcse-geography-lancaster.wikispaces.com/1.2+MIGRATION

Selected Migration Tool
59

Usage Details> Install ruby
> gem install mongify
… Modify the code to your needs
… Create configuration files
> mongify translation db.config >
translation.rb
> mongify process db.config translation.rb
60

Date Functions
• Year(), Month()… function included
• … buy only in the JavaScript engine
• Solution: New fields!
• [original field]
• [original field]_[year part]
• [original field]_[month part]
• [original field]_[day part]
• [original field]_[hour part]
61

NO SCHEMA IS A GOOD THING BUT…
Schemaless
62

Default Values
• No Schema
• No Default Values
• App Challenge
• Timestamps…
No single source of truth
63

Casting and Type Safety
• No Schema
• No …
• App Challenge
64

Auto Numbers
• Start using _id
{
"_id" : 0,
"health" : 1,
"stateStr" : "PRIMARY",
"uptime" : 59917
}
• Counter tables
• Dedicated database
• 1:1 Mapping
• Counter++ using findAndModify
65

ORM Solution
66

Data Analysts
67
http://www.designersplayground.com/pr/internet-meme-list/data-analyst-2/

Data Analysts
• This is not SQL
• There are no joins
• No perfect tools
68
Pentaho
RockMongoMongoVUE RoboMongo

No Joins
• Do in the application
• Leverage the power of NoSQL
69

Limited Resultset
70
• 16MB document size
• GridFS

Bottom Line
• Powerful tool
• Embrace the Challenge
• Schema-less limitations: counters, data types
• Tools for Data Scientists
• Data design
71

Billing Data Model

Design Model
• balances
• bills
• lines
• plans
• queue
• rates
• subscribers
• users

Specific fields
Select All
db.users.find(
{ },
{ user_id: 1, status: 1, _id: 0 }
)
1: Show; 0: don’t show

WHERE
• != “A” { $ne: "A" }
• > 25 { $gt: 25 }
• > 25 AND <= 50 { $gt: 25, $lte: 50 }
• Like ‘bc%’ /^bc/
• < 25 OR >= 50 { $or : [ { $lt: 25 }, { $gte : 50 } ] }

Join
• Wrong Place…
• Or Map Reduce

82
 db.article.aggregate(
 { $group : {
 _id : { author : "$author“, name : “$name” },
 docsPerAuthor : { $sum : 1 },
 viewsPerAuthor : { $sum : "$pageViews" }
 }}
 );
GROUP BY
< GROUP BY author, name
< SUM(pageViews)
< SUM(1) = N

83
db.Movie.aggregate([
{$match:
{SeriesType : "F", MovieID : {$in : arrMovies}}
},
{$project:
{MovieID: "$MovieID", SeriesType: "$SeriesType",
Genres: "$Genres"}
},
{$unwind : "$Genres" },
{$group : { _id : "$Genres" , count : { $sum : 1 } } },
{$sort : { count: -1 }}
GROUP BY
WHERE
Keep some fields
Genres is an array
Counting and sorting

Aggregation Framework Operators
Operator Description
$project Adding/Removing fields
$match WHERE
$redact Changes document based on Doc content/structure
$limit First N documents
$skip Skips N docs
$unwind Turns array into a multiple documents
$group Group
$sort Sort
$geoNear Geo spatial
$out Write Output to collection

85
db.posts.update(
{“comments.email”: ”b@c.com”},
{$set : {“comments.email”: ”d@c.com”}}
}
SET age = age + 3
• db.users.update(
• { status: "A" } ,
• { $inc: { age: 3 } },
• { multi: true }
• )
UPDATE

86
j = { name : "mongo" }
k = { x : 3 }
db.things.insert( j )
db.things.insert( k )
INSERT

87
db.users.remove(
{ status: "D" }
)
DELETE

88
Every operation on a document is atomic
Two Phase Commit implementation is up to
you
Atomic Transactions: Single Row

89
 Multiple documents at once
db.foo.update(
{ status : "A" , $isolated : 1 },
{ $inc : { count : 1 } },
{ multi: true }
)
 Disclaimers:
• Sharding is not supported
• Not all or nothing (no roll back on failure)
Atomic Transactions: $isolated

90
 t = db.transactions.findAndModify({
query: {
state: "initial“
},
update: {
$set: {
state: "pending"
},
$currentDate: { lastModified: true }
},
new: true
})
Atomic Transactions: findAndModify

91
 If it is about complex transactions.
 Simplify the case.
 or Consider keeping w/ RDBMS
Atomic Transactions: Bottom Line

92
 Failure and order:
• db.collection.initializeOrderedBulkOp()
• db.collection.initializeUnorderedBulkOp()
 1000 ops/bulk:
var bulk = db.items.initializeUnorderedBulkOp();
bulk.insert( { item: "abc123", defaultQty: 100, status: "A", points: 100 } );
bulk.insert( { item: "ijk123", defaultQty: 200, status: "A", points: 200 } );
bulk.insert( { item: "mop123", defaultQty: 0, status: "P", points: 0 } );
bulk.execute();
Bulk Operations

93
 Create a new project
 Get the Maven configuration for MongoDB Java Driver
• http://mongodb.github.io/mongo-java-driver/
Project Setup

94
List l = new ArrayList();
/**** Insert ****/
// create a document to store key and value
for (int i = 1; i < 1000000; ++i) {
Document document = new Document()
.append("name", "Moshe Kaplan")
.append("age", 36 + i)
.append("createdDate", new Date());
l.add(document);
}
table.insertMany(l);
Bulk Ops in Java

95
List<String> continentList = Arrays.asList(new String[]{"Africa", "Europe", "Asia"});
DBObject match = new BasicDBObject("$match", new BasicDBObject("continent.name", new BasicDBObject("$in",
continentList)));
DBObject projectFields = new BasicDBObject("continent.name", 1);
projectFields.put("area", 1);
projectFields.put("_id", 0);
DBObject project = new BasicDBObject("$project", projectFields );
DBObject groupFields = new BasicDBObject( "_id", "$continent.name");
groupFields.put("average", new BasicDBObject( "$avg", "$area"));
DBObject group = new BasicDBObject("$group", groupFields);
List agList = new ArrayList();
agList.add(match);
agList.add(project);
agList.add(group);
MongoCursor<Document> cursor = countries.aggregate(agList).iterator();
while (cursor.hasNext()) {
System.out.println(cursor.next());
}
Aggregation Framework in Java

Performance Tuning
Make a Change

MONGODB TUNING
97

journalCommitInterval = 300:
Write to disk: 2ms <= t <= 300ms
Default 100ms, increase to 300ms to save resources
Disk
The Journal
98
Memory
Journal Data
1 2

RAM Optimization:
dataSize + indexSize < RAM
99
OS
Data Index
Journal

PROFILING AND SLOW LOG
100

Profiling Configuration
• Enable:
• mongod --profile=1 --slowms=15
• db.setProfilingLevel([level] , [time])
• How much:
• 0 (none)  1 (slow queries only)  2 (all)
• 100ms: default
• Where:
• system.profile collection @ local db
101

Profiling Results Analysis
• Last 5 >1ms: show profile
• w/o commands:
db.system.profile.find( { op: { $ne : 'command' } } ).pretty()
• Specific database:
db.system.profile.find( { ns : 'mydb.test' } ).pretty()
• Slower than:
db.system.profile.find( { millis : { $gt : 5 } } ).pretty()
• Between dates:
db.system.profile.find({ts : {
$gt : new ISODate("2012-12-09T03:00:00Z") ,
$lt : new ISODate("2012-12-09T03:40:00Z")
}}).pretty()
102

Explain
> db.courses.find().explain();
{ "cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 11, “nscannedObjects" : 11, "nscanned" : 11,
"nscannedObjectsAllPlans" : 11, "nscannedAllPlans" : 11,
"scanAndOrder" : false, "indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 0,
"indexBounds" : {},
"server" : "primary.domain.com:27017"
}
103

INDEXES
104

Index Management
• Regular Index
• db.users.createIndex( { user_id: 1 } )
• db.users.ensureIndex( { user_id: 1 } )
• Multiple + DESC Index
• db.users.ensureIndex( { user_id: 1, age: -1 } )
• Sub Document Index
• db.users.ensureIndex( { address.zipcode: 1 } )
• Unique Index
• db.users.ensureIndex( { address.zipcode: 1 } , { unique : true } )
• List Indexes
• db.users.getIndexes()
• Drop Indexes
• db.users.dropIndex(“indexName”)
105

Known Index Issues
• Bound filter should be the last (in the index as well).
• BitMap Indexes not really working
• You should design your indexes carefully
106

Dex: The Index Analyzer
• Installation:
• sudo apt-get -y install python-pip
sudo pip install dex
• Running:
• dex [mongodb_uri] (-f <logfile_path> | -p) [<options>]
• dex -w -p -n "testdb.*" mongodb://127.0.0.1/testdb -f
/var/log/mongodb/mongod.log

mtools: Visualize and Analyze Logs

Capped Collections
• Fixed size collections
• Circular buffers like
• High throughput operations
• Order guarantee
db.createCollection("mycoll", {capped: true, size:100000})
db.cappedCollection.find().sort( { $natural: -1 } )
• Case studies:
• Logs
• Cache

TTL
• Remove Old Data Automatically
• db.log_events.createIndex(
{ "createdAt": 1 }, { expireAfterSeconds: 3600 }
)
• db.log_events.insert( {
"expireAt": new Date('July 22, 2013 14:00:00'),
"logEvent": 2,
"logMessage": "Success!“
} )

ENVIRONMENT TUNING
111

TTL
• # For SSD only
• blockdev --setra 16 /dev/sdb
• blockdev --setra 16 /dev/dm-2
• # For all cluser mongod & mongos
• for i in /sys/kernel/mm/*transparent_hugepage/enabled;
do echo never > $i; done
• for i in /sys/kernel/mm/*transparent_hugepage/defrag;
do echo never > $i; done

STATS &
SCHEMA DESIGN
113

Sparse Matrix? I don’t Think so
• mongostat
• > db.stats();
• > db.collectionname.stats();
• Fragmentation if storageSize/size > 2
• db.collectionanme.runCommand(“compact”)
• Padding (wrong design) if paddingFactor > 2
114

High Availability
Going Real Time

(Do Not) Master/Slave

• In mongo.conf
• # Replication Options
• replSet=myReplSet
• > rs.initiate()
• > rs.conf()
• > rs.add(“host:port")
• rs.reconfig()
Replication Set
117

• rs.addArb(“host:port")
• Also:
• Low Priority
• Hidden
• (Weighted) Voting
Arbiter
118

Show Status: rs.status();
• {"set" : “myReplSet", "date" : ISODate("2013-02-05T10:23:28Z"),
• "myState" : 1,
• "members" : [
• {
• "_id" : 0, "name"
: "primary.example.com:27017",
• "health" : 1, "state" :
1,
• "stateStr" : "PRIMARY",
"uptime" : 164545,
• "optime" : Timestamp(1359901753000, 1),
• "optimeDate" : ISODate("2013-02-
03T14:29:13Z"), "self" : true
• },
• {
• "_id" : 1, "name"

Replica Set Recovery
• Create a new mongod
• Either install a plain vanilla
• Or duplicate existing mongod (better)
• Connect to the system
• Use the previous machine IP
• Or change configuration to remove old and add new

Sharding and Scale out:
Make a big Change
Map Reduce and Aggregation

Secondary Read Enabling

The Strategy : Sharding

MongoDB Implementation

Step 1: Create a Config ReplicaSet
• mkdir /data/configdb
• mongod --configsvr --dbpath /data/configdb --port 27019

Step 2: Install Mongos
• mongos --configdb config01:27019, config02:27019, config03:27019

Step 3: Add Shards
• Connect a mongos
• Add Shard
• sh.addShard( "rs1/mongodb0.example.net:27017" )
• sh.addShard( "mongodb0.example.net:27017" )

Step 4: Enable Sharding
• sh.enableSharding("<database>")

Step 5: Sharding Colleciton
• sh.shardCollection("<database>.<collection>", shard-key-pattern)
• sh.shardCollection("records.people", { "zipcode": 1, "name": 1 } )
• Keys:
• High Cardinality to enable split
• Use common query field
• Use Compound indexes for sharding

BACKUP AND MONITORING
130

First Option – Single Server
Logical Backup Physical Backup
Method mongodump Point in time snapshot (using LVM tools) or disk image/copy
(using AWS or Azure “external” tools)
Pros Low costs Low costs
Cons • Downtime: Long;
• Duration: Long (slow backup since logical data needs to be
extracted);
• Performance impact: High (slows the disks and may stuck the
machine on heavy used machines);
• Data consistency: Intact;
• Differential: Supported;
• Sharding: Supported;
• Downtime: OS and/or infrastructure depended;
• Duration: Short (faster backup since only data blocks are
copied);
• Performance impact: Unknown (depends on OS and/or
infrastructure);
• Data consistency: Unknown state;
• Differential: Infrastructure depended;
• Sharding: Unsupported;
131
Sharding: is a type of database partitioning that separates very large databases
the into smaller, faster, more easily managed parts called data shards. The word
shard means a small part of a whole..

SECOND Option – REPLICA SET
Logical Backup Physical Backup
Method mongodump Stop slave and copy its disk
Pros • Downtime: None (backup is performed using Slave server –
Master server is always up);
• Duration: Not significant (backup is performed using Slave
server);
• Performance impact: None (backup is performed using Slave
server – Master server is not impacted);
• Data consistency: Intact;
• Differential: Supported;
• Sharding: Supported;
• Downtime: None
• Duration: Not significant
Cons Very high costs – requires two additional servers. A slave
server of the same type and size as the master server; and a
small arbiter server (used as a secondary verification for
Master server availability tests and “voting”).
• Costs: Requires a dedicated server per replica set
132

THIRD OPTION - MongoDB MMS
• Part of the MongoDB Enterprise Edition or as a Cloud Service
• The Cloud Service offer
• $50/month/node
• $2.5/GB/Month backup.
• A valid go to market way of MongoDB
for upsale
• MMS Features
• Point in time recovery
• Daily snapshots
• Detailed monitoring
• Alerts
133

How to Enable Incremental Backup
• In Backup
• Use the --oplog flag when doing mongodump
• Dump each hour the local.oplog collection
• In recovery
• mongorestore --oplogReplay
• applyOps to implement hourly dump
134

mongostat

mongotop

db.serverStatus()

db.stats() and db.collection.stats()

rs.status()

STORAGE ENGINES
140

MMAPv1

MongoDB 3.0 and WiredTiger
• MongoDB version 3.0 supports new storage engine
(WiredTiger):
• Disk Compression
• Heavy write
• Document level locking
• File per collection
• Server wide selection:
• config.yaml
• launch w/ --storageEngine = wiredTiger
142

MongoDB Pluggable Architecture

Engines Comparison

YAML Based Configuration
storage:
dbPath: "/var/lib/mongodbwt"
directoryPerDB: true
engine: "wiredTiger"
wiredTiger:
engineConfig:
cacheSizeGB: 16
journalCompressor: zlib
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
systemLog:
destination: file
path: "/var/log/mongodb/mongod.log"
logAppend: true
timeStampFormat: iso8601-local
processManagement:
fork: true
pidFilePath: "/var/run/mongodb.pid"
#security:
# keyFile: "/etc/mongo.key"
# authorization: "enabled"
replication:
replSetName: "arp0"

SECURITY
146

Providing Permissions
• use admin
db.createUser( {
user: "siteUserAdmin", pwd: "password",
roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
} )
• use records
db.createUser( {
user: "recordsUserAdmin", pwd: "password",
roles: [ { role: "userAdmin", db: "records" } ]
} )

Roles
Read
readWrite
dbAdmin
dbOwner
userAdmin
clusterAdmin, clusterManager, …
backup, restore
readAnyDatabase, readWriteAnyDatabase
root

Granular Actions
use admin
db.createRole(
role: "manageOpRole",
privileges: [
{ resource: { cluster: true }, actions: [ "killop", "inprog" ] },
{ resource: { db: "", collection: "" }, actions: [ "killCursors" ] }
],
roles: []
}
)

Thank You !
Moshe Kaplan
moshe.kaplan@brightaqua.com
054-2291978

MongoDB from Basics to Scale

Recommended

More Related Content

Similar to MongoDB from Basics to Scale (20)

More from Moshe Kaplan (20)

Recently uploaded (20)

MongoDB from Basics to Scale