MongoDB Days UK: Using MongoDB to Build a Fast and Scalable Content Repository Sponsored by Nuxeo
2 likes921 views
Presented by Alain Escaffre, Director of Product Management, Nuxeo Experience level: Beginner MongoDB can be used in the Nuxeo Platform as a replacement for traditional SQL databases. Nuxeo's content repository, which is the cornerstone of this open source software platform, can now completely rely on MongoDB for data storage. This presentation will explain the motivation for using MongoDB and will discuss different implementation strategies. In this session, you will learn more about the migrations to MongoDB and how we were able to achieve increased performance gains.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to MongoDB Days UK: Using MongoDB to Build a Fast and Scalable Content Repository Sponsored by Nuxeo (20)
Ad
More from MongoDB (20)
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=560&fit=bounds)
Recently uploaded (20)
MongoDB Days UK: Using MongoDB to Build a Fast and Scalable Content Repository Sponsored by Nuxeo
- 1. Using MongoDB to Build a Fast and Scalable Content Repository
- 2. Some Context What we Do and What Problems We Try to Solve
- 3. Nuxeo Platform • We provide a Platform that developers can use to build highly customised Content Applications • We provide components, and the tools to assemble them • The Platform is open source (https://github.com/nuxeo) • Various customers - various use cases • Me: Product Director at Nuxeo @aescaffre
- 4. Document Repository • Document Oriented Database ➡ store JSON documents • Document Repository ➡Manage Document attributes, hierarchy, blobs, security, lifecycle, versions
- 5. Document Repository Storage abstraction:be able to choose the right storage: • Depending on the constraints • Depending on the environment
- 6. Document Repository A Nuxeo Platform document
- 7. Document Repository With custom schemas:
- 8. Document Repository Security on each record:
- 9. Document Repository Thumbnail, Preview URLs Get A Conversion (image, video, office, sound) GET http://localhost:8080/nuxeo/api/v1/path/{docPath}/@convert?type=application%2Fpd
- 10. Document Repository •Start a Worklfow curl -X POST 'http://localhost:8080/nuxeo/site/automation/ Context.StartWorkflow' -H 'Accept: */*' -H 'Authorization: Basic QWRtaW5pc3RyYXRvcjpBZG1pbmlzdHJhdG9y’ -H 'content-type: application/json+nxrequest' -d '{"params":{"id":"serial- review","start":"true"},"input":"/default-domain/Passports/ 3719050812174596321","context":{}}' •Do Some Queries SELECT * FROM Document WHERE files/*1/file/name LIKE '%.txt' AND files/*1/file/length = 0
- 11. History : Nuxeo Repository And Storage • 2006: Nuxeo Repository is based on ZODB (Python / Zope based): • This is not JSON in NoSQL, but Python serialization in ObjectDB • Concurrency and performances issues, Bad transaction handling2007: Nuxeo Platform 5.1 - Apache JackRabbit (JCR based) • 2007: Nuxeo Platform 5.1 - Apache JackRabbit (JCR based) • Mix SQL + Java Serialization + Lucene • Transaction and consistency issues • 2009: Nuxeo 5.2 - Nuxeo VCS • SQL based repository : MVCC & ACID • very reliable, but some use cases can not fit in a SQL DB ! • 2014: Nuxeo 5.9 - Nuxeo DBS • Document Based Storage repository • MongoDB is the reference backend
- 12. From SQL to NoSQL Understanding the motivations for moving to MongoDB
- 14. KEY LIMITATIONS OF THE SQL APPROACH • Impedance Issues • storing Documents in tables is not easy • requires Caching and Lazy loading • Scalability • Document repository can become very large (versions, workflows …) • Scaling out SQL DB is very complex (and never transparent) • Concurrency model • Heavy write is an issue (Quotas, Inheritance) • Heavy write is an issue (Quotas, Inheritance)
- 15. Need a Different Storage Model!
- 16. From SQL to NoSQL
- 17. NoSQL with MongoDB • No Impedance Issue ‣ One Nuxeo Document = One MongoDB Document • No Scalability Issue ‣ Native distributed architecture allows scale out • No Concurrency Issue ‣ Document Level "Transactions" • No Application Level Cache is Needed ‣ No need to manage invalidations
- 18. MongoDB Integration Inside nuxeo-dbs storage adapter
- 19. Document base Storage & Mongodb
- 20. Storing Nuxeo Documents in MongoDB
- 21. Hierarchy • Parent-child relationship: ecm:parentId • Recursion optimised through ecm:ancestorIds array • Maintained by the framework (create, delete, move, copy)
- 22. Security • Generic ACP stored in ecm:acp field • Precomputed Read ACLs to avoid post-filtering on search • Simple Set of identities having access • Semantic restriction on blocking • Maintained by framework • Search matches if intersection
- 23. Search
- 24. Consistency Challenges • Unitary Document Operations Are Safe • No impedance issue • Large batch updates is not so much of an issue • SQL DB do not like long running transactions anyway • Multi-documents transactions are an issue • Workflows is a typical use case • Isolation issue • Other transactions can see intermediate states
- 25. Mitigating Consistency Issues • Transient State Manager • Run All Operations In Memory • Flush to MongoDB as late as possible • Populate an Undo log • Replay backward in case of Rollback ➡recover partial transaction management Complete isolation not possible • Need to flush transient state for queries • “uncommited” changes are visible to others ➡Read Uncommitted, at best
- 26. Typical Use Cases ForMongoDB
- 27. Huge Repository - Heavy Loading • Massive Amount of Documents (X00,000,000+ docs) ➡ Retail DAM repository, Banks archiving repository (email), large B2C companies invoicing output • Automatic and grape versioning: create a version for each single change ➡ Pharmaceutical,financial, etc.
- 28. Huge Repository - Heavy Loading • Massive Amount of Documents (X00,000,000+ docs) ➡ Retail DAM repository, Banks archiving repository (email), large B2C companies invoicing output • Automatic and grape versioning: create a version for each single change ➡ Pharmaceutical,financial, etc. SQL DB collapses (on commodity hardware) MongoDB handles the volume
- 30. Benchmarking Mass Import • Process 20000 documents ๏ 700 documents/s with SQL backend (cold cache) ๏ 6,000 documents/s with MongoDB / mmapv1: x9 ๏ 11,000 documents/s with MongoDB / wiredTiger: x15 • Process 100000 documents ๏ 750 documents/s with SQL backend (cold cache) ๏ 9,500 documents/s with MongoDB / mmapv1: x9 ๏ 11,500 documents/s with MongoDB / wiredTiger: x15 • Process 200000 documents ๏ 750 documents/s with SQL backend (cold cache) ๏ 14,000 documents/s with MongoDB / mmapv1: x9 ๏ 11,000 documents/s with MongoDB / wiredTiger: x15
- 31. Benchmarking Scale Out • 1 Nuxeo node + 1 MongoDB node • 1900 docs/s • MongoDB CPU is the bottleneck (800%) • 2 Nuxeo nodes + 1 MongoDB node • 1850 docs/s • MongoDB CPU is the bottleneck (800%) • 2 Nuxeo nodes + 2 MongoDB nodes • 3400 docs/s when using read preferences Adding one MongoDB node adds 80% throughput
- 33. A Real Life Exemple
- 34. Context • Who: US Network Carier • Goal: Provide VOD Services • Requirements: • store videos • manage metadata • manage workflows • generate thumbs • generate conversions • manage availability Nuxeo Platform as a videos repository
- 35. Challenges • Very Large Objects: • Lots of metadata (dublincore, ADI, ratings) • Massive Daily Updates • Updates On Rights and Availability • Need To Track All Changes • Prove what was the availability for a given date Lots of data + lots of updates ➡ db.createCollection(“myMovies”)
- 36. MongoDB Choice • They chose MongoDB • because they have a good use case for MongoDB • because they wanted to use MongoDB • change work habits (Open source, NoSQL) • doing a project with MongoDB is cool!!