Rapid and Scalable Development with MongoDB, PyMongo, and Ming

Rick Copeland @rick446
Arborian Consulting, LLC

http://www.flickr.com/photos/fazen/9079179/

- Get started with PyMongo

- Sprinkle in some Ming schemas

- ODM: When a dict just won’t do

>>>importpymongo
>>>conn = pymongo.Connection()
>>>conn
Connection('localhost', 27017)
>>>conn.test
Database(Connection('localhost', 27017), u'test')
>>>conn.test.foo
Collection(Database(Connection('localhost', 27017), u'test'), u'foo')
>>>conn['test-db']
Database(Connection('localhost', 27017), u'test-db')
>>>conn['test-db']['foo-collection']
Collection(Database(Connection('localhost', 27017), u'test-db'), u'foo-
collection')
>>>conn.test.foo.bar.baz
Collection(Database(Connection('localhost', 27017), u'test'), u'foo.bar.baz')

>>>db = conn.test
>>>id = db.foo.insert({'bar':1, 'baz':[ 1, 2, {'k':5} ] })
>>>id
ObjectId('4e712e21eb033009fa000000') Auto-Generated _id
>>>db.foo.find()
<pymongo.cursor.Cursor object at 0x29c7d50>
>>>list(db.foo.find()) Cursors are python
[{u'bar': 1, u'_id': ObjectId('4e712e21eb033009fa000000'), u'baz': [1, 2, {'k': 5}]}]
generators
>>>db.foo.update({'_id':id}, {'$set': { 'bar':2}})
>>>db.foo.find().next()
{u'bar': 2, u'_id': ObjectId('4e712e21eb033009fa000000'), u'baz': [1, 2, {'k': 5}]}
>>>db.foo.remove({'_id':id})
>>>list(db.foo.find())
[]
Remove uses same
query language as find()

>>>db.foo.insert([ dict(x=x) forxinrange(10) ])
[ObjectId('4e71313aeb033009fa00000b'), … ]
>>>list(db.foo.find({ 'x': {'$gt': 3} })) Range Query
[{u'x': 4, u'_id': ObjectId('4e71313aeb033009fa00000f')}, {u'x': 5, u'_id':
ObjectId('4e71313aeb033009fa000010')}, {u'x': 6, u'_id':
ObjectId('4e71313aeb033009fa000011')}, …]
>>>list(db.foo.find({ 'x': {'$gt': 3} }, { '_id':0 } ))
[{u'x': 4}, {u'x': 5}, {u'x': 6}, {u'x': 7}, {u'x': 8}, {u'x': 9}]
>>>list(db.foo.find({ 'x': {'$gt': 3} }, { '_id':0 } )
....skip(1).limit(2))
[{u'x': 5}, {u'x': 6}]
>>>db.foo.ensure_index([
Partial Retrieval
...('x’,pymongo.ASCENDING),('y’,pymongo.DESCENDING)])
u'x_1_y_-1’

Compound Indexes

One Rule (for now): Avoid Javascript

http://www.flickr.com/photos/lizjones/295567490/

 You gotta write Javascript (for now)

 It’s pretty slow (single-threaded JS engine) 
MongoDB 2.2 with New
 Javascript is used by Aggregation Framework
 $where in a query Coming Real Soon Now ™
 .group(key, condition, initial, reduce, finalize=None)
 .map_reduce(map, reduce, out, finalize=None, …)

 Sharding gives some parallelism with .map_reduce() (and
possibly ‘$where’). Otherwise you’re single threaded.

>>>importgridfs
>>>fs = gridfs.GridFS(db) Python context
>>>withfs.new_file() asfp: manager
... fp.write('The file')
...
>>>fp
<gridfs.grid_file.GridIn object at 0x2cae910>
>>>fp._id
ObjectId('4e727f64eb03300c0b000003') Retrieve file by _id
>>>fs.get(fp._id).read()
'The file'

 Arbitrary data can be stored in the ‘fp’ object – it’s
just a Document (but please put it in ‘fp.metadata’)
 Mime type, links to other docs, etc.

>>>file_id = fs.put('Moar data!', filename='foo.txt')
>>>fs.get_last_version('foo.txt').read()
'Moar data!’
Create file by
filename
>>>file_id = fs.put('Evenmoar data!', filename='foo.txt')
>>>fs.get_last_version('foo.txt').read()
'Even moar data!’
>>>fs.get_version('foo.txt', -2).read()
'Moar data!’
>>>fs.list()
“2nd from the last”
[u'foo.txt']
>>>fs.delete(fs.get_last_version('foo.txt')._id)
>>>fs.list()
[u'foo.txt']
>>>fs.delete(fs.get_last_version('foo.txt')._id)
>>>fs.list()
[]

 Your data has a schema
 Your database can define and enforce it
 It can live in your application (as with MongoDB)
 Nice to have the schema defined in one place in the code

 Sometimes you need a “migration”
 Changing the structure/meaning of fields
 Adding indexes, particularly unique indexes
 Sometimes lazy, sometimes eager

 “Unit of work:” Queuing up all your updates can be handy

Model Datastore
(schema) (database)

Session

>>>importming.datastore
>>>ds = ming.datastore.DataStore('mongodb://localhost:27017', database='test')
>>>ds.db Connection +
Database(Connection('localhost', 27017), u'test') Database
>>>session = ming.Session(ds)
>>>session.db
Database(Connection('localhost', 27017), u'test')
>>>ming.configure(**{ Optimized for config
... 'ming.main.master':'mongodb://localhost:27017', files
... 'ming.main.database':'test'})
>>>Session.by_name('main').db
Database(Connection(u'localhost', 27017), u'test')

http://www.flickr.com/photos/pictureclara/5333266789/

frommingimport schema, Field

WikiDoc= collection(‘wiki_page', session,
Field('_id', schema.ObjectId()),
Index created on
Field('title', str, index=True),
import
Field('text', str))

CommentDoc= collection(‘comment', session,
Field('_id', schema.ObjectId()),
Field('page_id', schema.ObjectId(), index=True),
Field('text', str))
Shorthand for
schema.String

frommingimport Document, Session, Field

classWikiDoc(Document):
class__mongometa__:
session=Session.by_name(’main')
name='wiki_page’
indexes=[ ('title') ]
title = Field(str)
text = Field(str)

 Old declarative syntax continues to exist and be
supported, but it’s not being actively improved
 Sometimes nice when you want additional
methods/attrs on your document class

>>>doc = WikiDoc(dict(title='Cats', text='I can hazcheezburger?'))
>>>doc.m.save()
>>>WikiDoc.m.find()
<ming.base.Cursor object at 0x2c2cd90>
>>>WikiDoc.m.find().all() Documents are dict
subclasses
[{'text': u'I can hazcheezburger?', '_id': ObjectId('4e727163eb03300c0b000001'),
'title': u'Cats'}]
>>>WikiDoc.m.find().one().text
u'Ican hazcheezburger?’
>>>doc = WikiDoc(dict(tietul='LOL', text='Invisible bicycle'))
>>>doc.m.save()
Traceback(most recent call last):
File "<stdin>", line 1,
… Exception pinpoints
ming.schema.Invalid: <class 'ming.metadata.Document<wiki_page>'>:
problem
Extra keys: set(['tietul'])

>>>ming.datastore.DataStore('mim://', database='test').db
mim.Database(test)

 MongoDB is (generally) fast
 … except when creating databases
 … particularly when you preallocate

 Unit tests like things to be isolated

 MIM gives you isolation at the expense of speed & scaling

frommingimport schema, Field
fromming.odmimport (mapper, Mapper, RelationProperty,
ForeignIdProperty)
Plain Old Python
WikiDoc= collection('wiki_page',session, … ) Classes
CommentDoc= collection(’comment’, session, … )

classWikiPage(object): pass Map classes to
classComment(object): pass collection + session
odmsession.mapper(WikiPage, WikiDoc, properties=dict(
comments=RelationProperty('WikiComment')))
odmsession.mapper(Comment, CommentDoc, properties=dict(
page_id=ForeignIdProperty('WikiPage'),
page=RelationProperty('WikiPage')))

“Relations”

classWikiPage(MappedClass):
class__mongometa__:
session = main_odm_session
name='wiki_page’
indexes = [ 'title']

_id = FieldProperty(S.ObjectId)
title = FieldProperty(str)
text = FieldProperty(str)
comments = RelationProperty(’Comment’)

 Session ODMSession
 My_collection.m… My_mapped_class.query…
 ODMSessionactually does stuff
 Track object identity
 Track object modifications
 Unit of work flushing all changes at once
>>>pg = WikiPage(title='MyPage', text='is here')
>>>session.db.wiki_page.count()
0
>>>main_orm_session.flush()
>>>session.db.wiki_page.count()
1

http://www.flickr.com/photos/39747297@N05/5229733647/

 Various plug points in the session
 before_flush
 after_flush

 Some uses
 Logging changes to sensitive data or for analytics
 Full-text search indexing
 “last modified” fields
 Performance instrumentation

 Various plug points in the mapper
 before_/after_:
 Insert
 Update
 Delete
 Remove

 Some uses
 Collection/model-specific logging (user creation, etc.)
 Anything you might want a SessionExtension for but
would rather do per-model

PyMongo
http://api.mongodb.org/python
Apache License

Ming
http://sf.net/projects/merciless/
MIT License

Rick Copeland @rick446
Arborian Consulting, LLC

Feedback? http://www.surveymonkey.com/s/5DLCYKN

http://www.flickr.com/photos/f-oxymoron/5005673112/

Rapid and Scalable Development with MongoDB, PyMongo, and Ming

Recommended

More Related Content

What's hot (20)

Similar to Rapid and Scalable Development with MongoDB, PyMongo, and Ming (20)

More from Rick Copeland (12)

Recently uploaded (20)

Rapid and Scalable Development with MongoDB, PyMongo, and Ming

Editor's Notes