Mongo db eveningschemadesign

Consulting Engineer, MongoDB
Prasoon Kumar
#MongoDBDays #MongoDBIndia
MongoDB Introduction;
Schema Design

MongoDB is a ___________ database
1.  Document
2.  Open source
3.  High performance
4.  Horizontally scalable
5.  Full featured

1. Document Database
•  Not for .PDF & .DOC ﬁles
•  A document is essentially an associative array
•  Document = JSON object
•  Document = PHP Array
•  Document = Python Dict
•  Document = Ruby Hash
•  etc

1. NoSQL Data Model
Key-Value
Store
Riak
Memcache
Project
Voldemort
Redis
BerkeleyDB
Document
Database
MongoDB
CouchDB
OrientDB
Column-Family
Stores
Amazon
SimpleDB
Cassandra
Hbase
Hypertable
Graph
Databases
Neo4J
FlockDB
OrientDB

2. Open Source
•  MongoDB is an open source project
•  On GitHub
•  Licensed under the AGPL
•  Started & sponsored by MongoDB Inc (formerly
known as 10gen)
•  Commercial licenses available
•  Contributions welcome

3. High Performance
•  Written in C++
•  Extensive use of memory-mapped ﬁles
i.e.read-through write-through memory caching.
•  Runs nearly everywhere
•  Data serialized as BSON (fast parsing)
•  Full support for primary & secondary indexes
•  Document model = less work

Better Data
Locality
3. Performance
In-Memory Caching In-Place
Updates

4. Scalability
Auto-Sharding
•  Increase capacity as you go
•  Commodity and cloud architectures
•  Improved operational simplicity and cost visibility

4. High Availability
•  Automated replication and failover
•  Multi-data center support
•  Improved operational simplicity (e.g., HW swaps)
•  Data durability and consistency

4. Scalability: MongoDB Architecture

5. Full Featured
•  Ad Hoc queries
•  Real time aggregation
•  Rich query capabilities
•  Strongly consistent
•  Geospatial features
•  Support for most programming languages
•  Flexible schema

$ tar –zxvf mongodb-osx-x86_64-2.6.0.tgz
$ cd mongodb-osx-i386-2.6.0/bin
$ mkdir –p /data/db
$ ./mongod
Running MongoDB

MacBook-Pro-:~ $ mongo
MongoDB shell version: 2.6.0
connecting to: test
> db.cms.insert({text: 'Welcome to MongoDB'})
> db.cms.find().pretty()
{
"_id" : ObjectId("51c34130fbd5d7261b4cdb55"),
"text" : "Welcome to MongoDB"
}
Mongo Shell

_id
•  _id is the primary key in MongoDB
•  Automatically indexed
•  Automatically created as an ObjectId if not provided
•  Any unique immutable value could be used

ObjectId
•  ObjectId is a special 12 byte value
•  Guaranteed to be unique across your cluster
•  ObjectId("50804d0bd94ccab2da652599")
|----ts-----||---mac---||-pid-||----inc-----|
4 3 2 3

Terminology
RDBMS MongoDB
Table, View ➜ Collection
Row ➜ Document
Index ➜ Index
Join ➜ Embedded Document
Foreign Key ➜ Reference
Partition ➜ Shard

First step in any application is
Determine your entities

Entities in our Blogging System
•  Users (post authors)
•  Article
•  Comments
•  Tags,Category
•  Interactions (views,clicks)

In a relational base app
We would start by doing schema
design

Typical (relational) ERD
User
·Name
·Email address
Category
·Name
·URL
Comment
·Comment
·Date
·Author
Article
·Name
·Slug
·Publish date
·Text
Tag
·Name
·URL

In a MongoDB based app
We start building our app
and let the schema evolve

MongoDB ERD
User
·Name
·Email address
Article
·Name
·Slug
·Publish date
·Text
·Author
Comment[]
·Comment
·Date
·Author
Tag[]
·Value
Category[]
·Value

Seek = 5+ ms
Read = really really
fast
Post
Author
Comment
Disk seeks and data locality

Post
Author
Comment
Comment
Comment
Comment
Comment
Disk seeks and data locality

Real applications are not
built in the shell

MongoDB has native
bindings for over 12
languages

Drivers & Ecosystem
Drivers
Support for the most popular
languages and frameworks
Frameworks
Morphia
MEAN Stack
Java
Python
Perl
Ruby

MongoDB Drivers
•  Ofﬁcial Support for 12 languages
•  Community drivers for tons more
•  Drivers connect to mongo servers
•  Drivers translate BSON into native types
•  mongo shell is not a driver,but works like one in some
ways
•  Installed using typical means (maven,npm,pecl,gem,
pip)

# Python dictionary (or object)
>>> article = { ‘title’ : ‘Schema design in MongoDB’,
‘author’ : ‘prasoonk’,
‘section’ : ‘schema’,
‘slug’ : ‘schema-design-in-mongodb’,
‘text’ : ‘Data in MongoDB has a flexible schema.
So, 2 documents needn’t have same structure.
It allows implicit schema to evolve.’,
‘date’ : datetime.utcnow(),
‘tags’ : [‘MongoDB’, ‘schema’] }
>>> db[‘articles’].insert(article)
Design schema.. In application code

>>> img_data = Binary(open(‘article_img.jpg’).read())
>>> article = { ‘title’ : ‘Schema evolutionin MongoDB’,
‘author’ : ‘mattbates’,
‘section’ : ‘schema’,
‘slug’ : ‘schema-evolution-in-mongodb’,
‘text’ : ‘MongoDb has dynamic schema. For good
performance, you would need an implicit
structure and indexes’,
‘tags’ : [‘MongoDB’, ‘schema’, ‘migration’],
‘headline_img’ : {
‘img’ : img_data,
‘caption’ : ‘A sample document at the shell’
}}
Let’s add a headline image

>>> article = { ‘title’ : ‘Favourite web application framework’,
‘author’ : ‘prasoonk’,
‘section’ : ‘web-dev’,
‘slug’ : ‘web-app-frameworks’,
‘gallery’ : [
{ ‘img_url’ : ‘http://x.com/45rty’, ‘caption’ : ‘Flask’, ..},
..
]
‘tags’ : [‘Python’, ‘web’],
}
>>> db[‘articles’].insert(article)
And different types of article

>>> user = {
'user' : 'prasoonk',
'email' : 'prasoon.kumar@mongodb.com',
'password' : 'prasoon101',
'joined' : datetime.utcnow(),
'location' : { 'city' : 'Mumbai' },
}
} >>> db[‘users’].insert(user)
Users and proﬁles

Retrive using Comparison Operators
$gt,$gte,$in,$lt,$lte,$ne,$nin
•  Use to query documents
•  Logical:$or,$and,$not,$nor Element:$exists,$type
•  Evaluation:$mod,$regex,$where Geospatial:$geoWithin,$geoIntersects,$near,$nearSphere

Modelling comments (1)
•  Two collections–articles and comments
•  Use a reference (i.e. foreign key) to link together
•  But.. N+1 queries to retrieve article and comments
{
‘_id’: ObjectId(..),
‘title’:‘Schema design in MongoDB’,
‘author’:‘mattbates’,
‘date’: ISODate(..),
‘tags’: [‘MongoDB’, ‘schema’],
‘section’:‘schema’,
‘slug’:‘schema-design-in-mongodb’,
‘comments’:[ObjectId(..),…]
}
{ ‘_id’: ObjectId(..),
‘article_id’: 1,
‘text’: ‘A great article,helped me
understand schema design’,
‘date’: ISODate(..),,
‘author’:‘johnsmith’
}

$gt,$gte,$in,$lt,$lte,$ne,$nin
•  Use to query documents
•  Evaluation:$mod,$regex,$where Geospatial:$geoWithin,$geoIntersects,$near,$nearSphere
Comparison Operators
db.articles.find( { 'title' : ’Intro to MongoDB’ } )
db.articles.find( { ’date' : { ‘$lt’ :
{ISODate("2014-02-19T00:00:00.000Z") }} )
db.articles.find( { ‘tags’ : { ‘$in’ : [‘nosql’,‘database’] } } );

•  Single articles collection–
embed comments in article
documents
•  Pros
•  Single query, document designed
for the access pattern
•  Locality (disk, shard)
•  Cons
•  Comments array is unbounded;
documents will grow in size
(remember 16MB document
limit)
{
‘title’:‘Schema design in MongoDB’,
‘tags’: [‘MongoDB’,‘schema’],
…
‘comments’:[
{
‘text’: ‘Agreatarticle,helpedme
understandschemadesign’,
‘date’:ISODate(..),
},
…
]
}

•  Another option: hybrid of (2) and (3),embed
top x comments (e.g.by date,popularity) into
the article document
•  Fixed-size (2.4 feature) comments array
•  All other comments ‘overﬂow’ into a comments
collection (double write) in buckets
•  Pros
–  Document size is more ﬁxed – fewer moves
–  Single query built
–  Full comment history with rich query/aggregation

{
‘_id’:ObjectId(..),
‘title’:‘SchemadesigninMongoDB’,
‘tags’:[‘MongoDB’, ‘schema’],
…
‘comments_count’:45,
‘comments_pages’:1
‘comments’:[
{
‘text’: ‘Agreatarticle,helpedme
understandschemadesign’,
},
…
]
}
Total number of comments
•  Integer counter updated by
update operation as
comments added/removed
Number of pages
•  Page is a bucket of 100
comments (see next slide..)
Fixed-size comments array
•  10 most recent
•  Sorted by date on insertion

{
‘article_id’: ObjectId(..),
‘page’: 1,
‘count’: 42
‘comments’: [
{
‘text’: ‘A great article,helped me
understand schema design’,
},
…
}
One comment bucket
(page) document
containing up to about 100
comments
Array of 100 comment sub-
documents

Modelling interactions
•  Interactions
–  Article views
–  Comments
–  (Social media sharing)
•  Requirements
–  Time series
–  Pre-aggregated in preparation for analytics

Modelling interactions
•  Document per article per day–
‘bucketing’
•  Daily counter and hourly sub-
document counters for interactions
•  Bounded array (24 hours)
•  Single query to retrieve daily article
interactions; ready-made for
graphing and further aggregation
{
‘article_id’: ObjectId(..),
‘section’:‘schema’,
‘daily’: {‘views’: 45,‘comments’: 150 }
‘hours’: {
0 : {‘views’: 10 },
1 : {‘views’: 2 },
…
23 : {‘comments’: 14,‘views’: 10 }
}
}

JSON and RESTful API
Client-side
JSON
(eg AngularJS, (BSON)
Real applications are not built at a shell–let’s build a RESTful API.
Pymongo driver
Python web
app
HTTP(S) REST
Examples to follow: Python RESTful API using Flask microframework

myCMS REST endpoints
Method URI Action
GET /articles Retrieve all articles
GET /articles-by-tag/[tag] Retrieve all articles by tag
GET /articles/[article_id] Retrieve a specific article by article_id
POST /articles Add a new article
GET /articles/[article_id]/comments Retrieve all article comments by
article_id
POST /articles/[article_id]/comments Add a new comment to an article.
POST /users Register a user user
GET /users/[username] Retrieve user’s profile
PUT /users/[username] Update a user’s profile

$ git clone http://www.github.com/prasoonk/mycms_mongodb
$ cd mycms-mongodb
$ virtualenv venv
$ source venv/bin/activate
$ pip install –r requirements.txt
$ mkdir –p data/db
$ mongod --dbpath=data/db --fork --logpath=mongod.log
$ python web.py
[$ deactivate]
Getting started with the skeleton code

@app.route('/cms/api/v1.0/articles', methods=['GET'])
def get_articles():
"""Retrieves all articles in the collection
sorted by date
"""
# query all articles and return a cursor sorted by date
cur = db['articles'].find().sort('date’)
if not cur:
abort(400)
# iterate the cursor and add docs to a dict
articles = [article for article in cur]
return jsonify({'articles' : json.dumps(articles, default=json_util.default)})
RESTful API methods in Python + Flask

@app.route('/cms/api/v1.0/articles/<string:article_id>/comments', methods = ['POST'])
def add_comment(article_id):
"""Adds a comment to the specified article and a
bucket, as well as updating a view counter
"””
…
page_id = article['last_comment_id'] // 100
…
# push the comment to the latest bucket and $inc the count
page = db['comments'].find_and_modify(
{ 'article_id' : ObjectId(article_id),
'page' : page_id},
{ '$inc' : { 'count' : 1 },
'$push' : {
'comments' : comment } },
fields= {'count' : 1},
upsert=True,
new=True)

# $inc the page count if bucket size (100) is exceeded
if page['count'] > 100:
db.articles.update(
{ '_id' : article_id,
'comments_pages': article['comments_pages'] },
{ '$inc': { 'comments_pages': 1 } } )
# let's also add to the article itself
# most recent 10 comments only
res = db['articles'].update(
{'_id' : ObjectId(article_id)},
{'$push' : {'comments' : { '$each' : [comment],
'$sort' : {’date' : 1 },
'$slice' : -10}},
'$inc' : {'comment_count' : 1}})
…

def add_interaction(article_id, type):
"""Record the interaction (view/comment) for the
specified article into the daily bucket and
update an hourly counter
"""
ts = datetime.datetime.utcnow()
# $inc daily and hourly view counters in day/article stats bucket
# note the unacknowledged w=0 write concern for performance
db['interactions'].update(
{ 'article_id' : ObjectId(article_id),
'date' : datetime.datetime(ts.year, ts.month, ts.day)},
{ '$inc' : {
'daily.{}’.format(type) : 1,
'hourly.{}.{}'.format(ts.hour, type) : 1
}},
upsert=True,
w=0)

$ curl -i http://localhost:5000/cms/api/v1.0/articles
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 335
Server: Werkzeug/0.9.4 Python/2.7.5
Date: Thu, 10 Apr 2014 16:00:51 GMT
{
"articles": "[{"title": "Schema design in MongoDB", "text": "Data in MongoDB
has a flexible schema..", "section": "schema", "author": "prasoonk", "date":
{"$date": 1397145312505}, "_id": {"$oid": "5346bef5f2610c064a36a793"},
"slug": "schema-design-in-mongodb", "tags": ["MongoDB", "schema"]}]"}
Testing the API – retrieve articles

$ curl -H "Content-Type: application/json" -X POST -d '{"text":"An interesting
article and a great read."}'
http://localhost:5000/cms/api/v1.0/articles/52ed73a30bd031362b3c6bb3/
comments
{
"comment": "{"date": {"$date": 1391639269724}, "text": "An interesting
article and a great read."}”
}

Testing the API – comment on an article

Schema iteration
New feature in the backlog?
Documents have dynamic schema so we just iterate the
object schema.
>>> user = {‘username’:‘matt’,
‘ﬁrst’:‘Matt’,
‘last’:‘Bates’,
‘preferences’: {‘opt_out’: True } }
>>> user.save(user)

Further reading
•  ‘myCMS’ skeleton source code:
http://www.github.com/prasoonk/mycms_mongodb
•  Data Models
http://docs.mongodb.org/manual/data-modeling/
•  Use case-metadata and asset management:
http://docs.mongodb.org/ecosystem/use-cases/metadata-and-
asset-management/
•  Use case-storing comments:
http://docs.mongodb.org/ecosystem/use-cases/storing-
comments/

Online Training at MongoDB University

For More Information
Resource Location
MongoDB Downloads mongodb.com/download
Free Online Training education.mongodb.com
Webinars and Events mongodb.com/events
White Papers mongodb.com/white-papers
Case Studies mongodb.com/customers
Presentations mongodb.com/presentations
Documentation docs.mongodb.org
Additional Info info@mongodb.com
Resource Location

We've introduced a lot of
concepts here

Schema Design @
User
·Name
·Email address
Article
·Name
·Slug
·Publish date
·Text
·Author
Comment[]
·Comment
·Date
·Author
Tag[]
·Value
Category[]
·Value

Replication @
Secondary Secondary
Primary
Client Application
Driver
Write
Read
Read

Indexing @
7 16
1 2 5 6 9 12 18 21

Sharding @
www.etiennemansard.com

Consulting Engineer, MongoDB
Prasoon Kumar
#MongoDBDays #MongoDBIndia @prasoonk
Thank You

Mongo db eveningschemadesign

More Related Content

What's hot (20)

Similar to Mongo db eveningschemadesign (20)

Recently uploaded (20)

Mongo db eveningschemadesign