MongoDB .local Bengaluru 2019: The Journey of Migration from Oracle to MongoDB at Rakuten
1 like513 views
Find out more about our journey of migrating to MongoDB after using Oracle for our hotel search database for over ten years. - How did we solve the synchronization problem with the Master Database? - How to get fast search results (even with massive write operations)? - How other issues were solved
Ad
More Related Content
What's hot (20)
Similar to MongoDB .local Bengaluru 2019: The Journey of Migration from Oracle to MongoDB at Rakuten (20)
Ad
More from MongoDB (20)
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
MongoDB .local Bengaluru 2019: The Journey of Migration from Oracle to MongoDB at Rakuten
- 2. Our journey of Migration from the legacy datastore to MongoDB Akihiro Tatematsu, Assistant Manager, Rakuten, Inc.
- 3. About me 2010- • Oracle DBA Exadata, RAC, TimesTen, etc 2016- • NoSQL DBA MongoDB, Elasticsearch, Redis, etc Akihiro Tatematsu
- 4. Contents Introduction Selection of Next Datastore Optimization Summary
- 5. Introduction
- 8. About Rakuten Total Global GTV (JPY tn) Rakuten (2019.05.14) https://global.rakuten.com/corp/investors/documents/results/2018.html
- 10. About Rakuten Travel Rakuten Travel (2019.05.14) https://travel.rakuten.com/
- 11. MongoDB in Rakuten Travel Usage service 19services Total Number of operations 260k/sec Number of mongod 111nodes MongoDB for Search 15nodes
- 12. Necessity of changing Datastore Serviceability License Cost Scalability Data Sync Labor Cost
- 13. Selection of Next Datastore
- 14. Our Search system We used TimesTen as a cache server in 2009 Search Engine Front Service Search Engine Search Engine Push the same data each cluster Oracle TimesTenOracle
- 15. Traffic Analysis and Forecast License Cost No Horizontal Scale-Out Duplicated Data 2013 2014 2015 2016 2017 Search Query (TPS) 2013 2014 2015 2016 2017 Data Size (GB)
- 16. Verifications by POC Low Latency Flexibility Cost Efficiency SLA 99.95% Horizontal Scale Out Trouble Shooting Fault Tolerance MongoDB Oracle Coherence Cassandra Redis MEMSQL
- 17. New challenge project Movement to Microservice Architecture Improvement of Applications Performance Improvement of User Interface Develop search APIs Synchronize Data from Oracle to MongoDB
- 18. Data Synchronization How we synchronize data into MongoDB Oracle Kafka Consumers MongoDB Sync Tables High throughput Low latency Scalability Centralized Real-time Operate hundreds of Instances Write 500 million messages a day Avg. 7ms per 10 message
- 19. Let’s use MongoDB Do you know how to use MongoDB? We can be an expert in MongoDB
- 20. Search MongoDB environment Monitoring Grafana + Prometheus Backup LVM snapshot + oplog Log management logstash, logrotated, mtools Enterprise function Audit, OpsManager Config management Bitbucket + Confluence tools Physical mongod Physical mongod Physical mongod Routers VM mongos VM mongos VM mongos Config Servers VM mongod VM mongod VM mongod VM mongos API VM mongos API VM API VM API Shard(x5)
- 21. What have we been through 2,000 QPS Business Expected Load 8,000 QPS Designed Load 100,000 QPS Actual Load 1250% 3120%
- 22. Problems with Mass Operations 170kops
- 23. Optimization
- 24. Search engineer‘s jurisdiction Oracle Kafka Consumers MongoDB Provider User Mainly Writes Mainly Reads
- 25. What have we been through Release Retryable Writes setting Bulk Write operation Write Concern setting Optimize Over 14 Improvement
- 26. Retryable Writes Consumers Primary Secondary Write API services Read Secondary Consumers Primary Secondary à Primary Write API services Read Down 1 ~ 2 seconds Secondary com.mongodb.MongoWriteException: not master
- 27. Retryable Writes retryWrites : true After retrying then send message to new master Response time to write Consumers Primary Secondary à Primary Write API services Read Down Secondary
- 28. Bulk Write operation MongoDB PIC1 PIC2 PIC3 Media Table Update sort order Oracle Collection 1 Collection 2db.collection.insert() db.collection.update()
- 29. Bulk Write operation MongoDB PIC1 PIC2 PIC3 Media Table Update sort order Oracle Collection 1 Collection 2db.collection.insert() db.collection.bulkWrite() Group by same collection messages
- 31. Write Concern settings (w) w Option The w option requests acknowledgment that the write operation has propagated to a specified number of mongod instances or to mongod instances with specified tags. Our initial setting w : 1 (default) Driver Primary Secondary Secondary write Apply Replication w:0 w:1 w:2 (majority) Response Case of 3 nodes replica set Apply
- 32. Write Concern settings (w : 0) We have to do something. Can we set w=0? APP DBA If you want to take that risk, sure
- 33. Trouble (w : 0) Trouble Primary nodes were downed by high load.
- 34. Trouble (w : 0) Driver Primary Secondary Secondary write ApplyResponse 'Dirty cache' size grew much larger WiredTigerLAS.wt became too big ("LAS" == lookaside, a.k.a WT cache overflow.) Apply
- 35. Trouble (w : 0) Dirty Bytes 150GB
- 37. Write Concern settings (j : false) journal memory disk When w: <number> default of j is false (Acknowledgment requires writing operation in memory) j Option The j option requests acknowledgment from MongoDB that the write operation has been written to the on-disk journal. Our initial setting j : true 100ms Writes (j:true) Force sync Writes (j:false) writes Journal Commit
- 38. Write Concern settings (j : false) Before After Improvement 64% 4.96 ms 1.79 ms
- 39. Reads operations Operations Per Shard 25kops Read latency 10ms
- 40. New features of MongoDB 4.0 Multi- Document ACID Transactions Aggregation Pipeline Type Conversions Non-Blocking Secondary Reads 40% Faster Data Migrations Extensions to Change Streams and so forth MongoDB (2019.05.14) https://www.mongodb.com/blog/post/secondary-reads-mongodb-40 Multi- Document ACID Transactions Aggregation Pipeline Type Conversions Non-Blocking Secondary Reads 40% Faster Data Migrations Extensions to Change Streams and so forth
- 41. Upgrade to version 4 Ver3.6 Ver4.0 Operations Read latency
- 42. Current situation Operations Per Shard 30kops Writes Latency 1.5ms Reads Latency 0.2ms Reads Latency (max) 1.3ms
- 43. Summary
- 44. Next Datastore Oracle TimesTen Oracle Golden Gate Old architecture New architecture Oracle kafka Consumers MongoDB Serviceability License Cost Scalability Data Sync Labor Cost MongoDB Oracle Coherence Cassandra Redis MEMSQL
- 45. Optimization Operations Writes Latency Reads Latency Write concern settings w : 1, j : false (default) Latency Writes : 1.5ms Reads : 0.2ms MongoDB version 4.0 (latest) Retryable Writes retryWrites=true Bulk Write Operation
- 46. Future outlook DBA APP ORACLE redis Azure Google Cloud Platform Elasticsearch
- 47. Thank You!