Mongodb - Scaling write performance

Oct 15, 201248 likes15,628 views

- MongoDB allows for automatic sharding of data across multiple servers to improve write performance. However, scaling write performance is challenging due to the way B-tree indexes handle random inserts. - To improve write performance, one can partition data by time or use a hash shard key. However, these have limitations as the data grows large. The best approach is to use a low-cardinality hash prefix combined with a sequential part for the shard key. - Proper choice of shard key is crucial for scaling MongoDB's write performance as data size increases. Linear scalability is difficult to achieve and alternative databases may be better if extremely high write throughput is required.

MongoDB:
Scaling write performance
Junegunn Choi

MongoDB

• Document data store

• JSON-like document

• Secondary indexes

• Automatic failover

• Automatic sharding

First impression:
Easy

• Easy installation

• Easy data model

• No prior schema design

• Native support for secondary indexes

Second thought:
Not so easy

• No SQL

• Coping with massive data growth

• Setting up and operating sharded cluster

• Scaling write performance

Today we’ll talk about
insert performance

Steady 5k inserts/sec

* 1kB record. ObjectId as PK
* WriteConcern: Journal sync on Majority

Insert throughput
on a replica set
with a secondary index

Culprit:
B+Tree index

• Good at sequential insert

• e.g. ObjectId, Sequence #, Timestamp

• Poor at random insert

• Indexes on randomly-distributed data

Sequential vs. Random insert
1 55
2 75
3 78
4 1
5 99
6 36
7 80
8 91
9 52
10 B+Tree 63 B+Tree
11 56
12 33
working set working set

Sequential insert ➔ Small working set Random insert ➔ Large working set
➔ Fits in RAM ➔ Sequential I/O ➔ Cannot ﬁt in RAM ➔ Random I/O
(bandwidth-bound) (IOPS-bound)

1. Partitioning

Aug 2012 Sep 2012 Oct 2012

B+Tree

ﬁts in memory

does not ﬁt in memory

1. Partitioning

• MongoDB doesn’t support partitioning

• Partitioning at application-level

• e.g. Daily log collection

• logs_20121012

2. Better H/W

• More RAM

• More IOPS

• RAID striping

• SSD

• AWS Provisioned IOPS (1k ~ 10k)

3. More H/W: Sharding
• Automatic partitioning across nodes

SHARD1 SHARD2 SHARD3

mongos router

There’s no free lunch
• Manual partitioning

• Incidental complexity

• Better H/W

• $

• Sharding

• $$

• Operational complexity

Scaling insert performance
with sharding

Shard key example:
year_of_birth
64MB chunk

~ 1950 1971 ~ 1990 1951 ~ 1970

1991 ~ 2005 2006 ~ 2010

2010 ~ ∞

USERS USERS USERS
SHARD1 SHARD2 SHARD3

mongos router

Sequential key

• ObjectId as shard key

• Sequence #

• Timestamp

Sequential key

1000 ~ 2000
• All inserts into one chunk
5000 ~ 7500
• Cannot scale insert performance
9000 ~ ∞
• Chunk migration overhead USERS
SHARD-x

9001, 9002, 9003, 9004, ...

Hash key

• e.g. SHA1(_id) = 9f2feb0f1ef425b292f2f94 ...

• Distributes inserts evenly across all chunks

Hash key

• Performance drops as collection grows

• Why? Mandatory index on shard key

• B+Tree problem again!

Sequential + hash key
• Coarse-grained sequential preﬁx

• e.g. Year-month + hash value

• 201210_24c3a5b9

B+Tree

201208_* 201209_* 201210_*

But what if...

B+Tree

large working set

201208_* 201209_* 201210_*

Sequential + hash key

• Can you predict data growth rate?

• Balancer not clever enough

• Only considers # of chunks

• Migration slow during heavy-writes

Sequential key
Hash key
Sequential + hash key

Low-cardinality hash key
• Small portion of hash value Shard key range: A ~ D

• e.g. A~Z, 00~FF

• Alleviates B+Tree problem
Local
• Sequential access on ﬁxed # B+Tree
of parts

• Cardinality / # of shards
A A A B B B C C C

Low-cardinality hash key

• Limits the # of possible chunks

• e.g. 00 ~ FF ➔ 256 chunks

• Chunk grows past 64MB

• Balancing becomes difﬁcult

Sequential key
Hash key
Sequential + hash key
Low-cardinality hash key

Low-cardinality hash preﬁx
+ sequential part
Shard key range: A000 ~ C999

• e.g. Short hash preﬁx + timestamp

• Nice index access pattern Local
B+Tree
• Unlimited number of chunks

A000 A123 B000 B123 C000 C123

Lessons learned
• Know the performance impact of secondary index

• Choose the right shard key

• Test with large data sets

• Linear scalability is hard

• If you really need it, consider HBase or Cassandra

• SSD

Thank you. Questions?

유응섭 rspeed@daumcorp.com
최준건 gunn@daumcorp.com

MariaDB 10.0 introduces domain-based parallel replication which allows transactions in different domains to execute concurrently on replicas. This can result in out-of-order transaction commit. MariaDB 10.1 adds optimistic parallel replication which maintains commit order. The document discusses various parallel replication techniques in MySQL and MariaDB including schema-based replication in MySQL 5.6 and logical clock replication in MySQL 5.7. It provides performance benchmarks of these techniques from Booking.com's database environments.

Sharding Methods for MongoDBMongoDB

Sharding in MongoDB allows for horizontal scaling of data and operations across multiple servers. When determining if sharding is needed, factors like available storage, query throughput, and response latency on a single server are considered. The number of shards can be calculated based on total required storage, working memory size, and input/output operations per second across servers. Different types of sharding include range, tag-aware, and hashed sharding. Choosing a high cardinality shard key that matches query patterns is important for performance. Reasons to shard include scaling to large data volumes and query loads, enabling local writes in a globally distributed deployment, and improving backup and restore times.

InnoDB MVCC Architecture (by 권건우)I Goo Lee.

1. MySQL uses Multi-Version Concurrency Control (MVCC) to allow for concurrent reads and writes to the same data. 2. MVCC works by storing the data in multiple versions and using read views and undo logs to present different versions of the data to transactions based on their isolation level and start time. 3. The InnoDB storage engine in MySQL implements MVCC using undo logs to reconstruct past versions of rows from writes and maintain multiple read views with different views of the data for concurrent transactions.

Introduction to RedisDvir Volk

Introduction to memcachedJurriaan Persyn

MyRocks Deep DiveYoshinori Matsunobu

RocksDB Performance and Reliability PracticesYoshinori Matsunobu

Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale. In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.

Salvatore Sanfilippo – How Redis Cluster works, and why - NoSQL matters Barce...NoSQLmatters

Salvatore Sanfilippo – How Redis Cluster works, and why In this talk the algorithmic details of Redis Cluster will be exposed in order to show what were the design tensions in the clustered version of an high performance database supporting complex data type, the selected tradeoffs, and their effect on the availability and consistency of the resulting solution.Other non-chosen solutions in the design space will be illustrated for completeness.

Kafka replication apachecon_2013Jun Rao

The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.

Cassandra Introduction & FeaturesDataStax Academy

MongoDB Administration 101MongoDB

The document provides an overview of MongoDB administration including its data model, replication for high availability, sharding for scalability, deployment architectures, operations, security features, and resources for operations teams. The key topics covered are the flexible document data model, replication using replica sets for high availability, scaling out through sharding of data across multiple servers, and different deployment architectures including single/multi data center configurations.

Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné

Of course there is no such thing as perfect service discovery, and we will see why in the talk. However, the way ProxySQL is deployed in this case minimizes the risk for split-brains, and this is why I qualify it as almost perfect. But let’s step back a little... MySQL alone is not a high availability solution. To provide resilience to primary failure, other components need to be integrated with MySQL. At MessageBird, these additional components are ProxySQL and Orchestrator. In this talk, we describe how ProxySQL is architectured to provide close to perfect Service Discovery and how this, combined with Orchestrator, allows for automatic failover. The talk presents the details of the integration of MySQL, ProxySQL and Orchestrator in Google Cloud (and it would be easy to re-implement a similar architecture at other cloud vendors or on-premises). We will also cover lessons learned for the 2 years this architecture has been in production. Come to this talk to learn more about MySQL high availability, ProxySQL and Orchestrator.

MySQL_SQL_Tunning_v0.1.3.docxNeoClova

RocksDB detailMIJIN AN

RocksDB is an embedded key-value store written in C++ and optimized for fast storage environments like flash or RAM. It uses a log-structured merge tree to store data by writing new data sequentially to an in-memory log and memtable, periodically flushing the memtable to disk in sorted SSTables. It reads from the memtable and SSTables, and performs background compaction to merge SSTables and remove overwritten data. RocksDB supports two compaction styles - level style, which stores SSTables in multiple levels sorted by age, and universal style, which stores all SSTables in level 0 sorted by time.

Introduction to MongoDBMike Dirolf

MongoDB is an open-source, document-oriented database that provides high performance and horizontal scalability. It uses a document-model where data is organized in flexible, JSON-like documents rather than rigidly defined rows and tables. Documents can contain multiple types of nested objects and arrays. MongoDB is best suited for applications that need to store large amounts of unstructured or semi-structured data and benefit from horizontal scalability and high performance.

Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive

This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.

RocksDB compactionMIJIN AN

The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.

MongoDB WiredTiger Internals: Journey To TransactionsMydbops

MongoDB has adapted transaction feature (ACID Properties) in MongoDB 4.0. This talk focuses on the internals of how MongoDB adapted the ACID properties with Weird Tiger Engine. Weird tiger offers more future possibilities for MongoDB. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Manosh Malai Senior Devops/NoSQL Consultant with Mydbops and Ranjith Database Administrator with Mydbops.

mongodb와 mysql의 CRUD 연산의 성능 비교Woo Yeong Choi

How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015PostgreSQL-Consulting

This document discusses how PostgreSQL works with disks and provides recommendations for disk subsystem monitoring, hardware selection, and configuration tuning to optimize performance. It explains that PostgreSQL relies on disk I/O for reading pages, writing the write-ahead log (WAL), and checkpointing. It recommends monitoring disk utilization, IOPS, latency, and I/O wait. The document also provides tips for choosing hardware like SSDs or RAID configurations and configuring the operating system, file systems, and PostgreSQL to improve performance.

MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...Jean-François Gagné

Since 5.7.2, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK (DATABASE based parallel replication is also implemented in 5.6 but this is not covered in this talk). In early 5.7 versions, parallel replication was based on group commit (like MariaDB) and 5.7.6 changed that to intervals. Intervals are more complicated but they are also more powerful. In this talk, I will explain in detail how they work and why intervals are better than group commit. I will also cover how to optimize parallel replication in MySQL 5.7 and what improvements are coming in MySQL 8.0. I will also explain why Group Replication is replicating faster than standard asynchronous replication. Come to this talk to get all the details about MySQL 5.7 Parallel Replication.

Redisimalik8088

Redis is an in-memory key-value database that is used by companies like Instagram and Stack Overflow for caching and storing non-relational data. It supports common data structures like strings, hashes, lists, sets and sorted sets. Redis provides very fast performance of over 100,000 writes and 80,000 reads per second. While Redis does not support complex queries, it is well suited for simple, fast queries and retrievals. The document discusses how Redis could be used to query and retrieve flight booking data by airport in both coarse and fine-grained data modeling approaches.

InnoDB Locking Explained with Stick FiguresKarwin Software Solutions LLC

When does InnoDB lock a row? Multiple rows? Why would it lock a gap? How do transactions affect these scenarios? Locking is one of the more opaque features of MySQL, but it’s very important for both developers and DBA’s to understand if they want their applications to work with high performance and concurrency. This is a creative presentation to illustrate the scenarios for locking in InnoDB and make these scenarios easier to visualize. I'll cover: key locks, table locks, gap locks, shared locks, exclusive locks, intention locks, insert locks, auto-inc locks, and also conditions for deadlocks.

Parallel Replication in MySQL and MariaDBMydbops

A Technical Introduction to WiredTigerMongoDB

This document provides an overview of WiredTiger, an open-source embedded database engine that provides high performance through its in-memory architecture, record-level concurrency control using multi-version concurrency control (MVCC), and compression techniques. It is used as the storage engine for MongoDB and supports key-value data with a schema layer and indexing. The document discusses WiredTiger's architecture, in-memory structures, concurrency control, compression, durability through write-ahead logging, and potential future features including encryption and advanced transactions.

InnoDB Performance OptimisationMydbops

Introduction to Storm Chandler Huang

Storm is a distributed and fault-tolerant realtime computation system. It was created at BackType/Twitter to analyze tweets, links, and users on Twitter in realtime. Storm provides scalability, reliability, and ease of programming. It uses components like Zookeeper, ØMQ, and Thrift. A Storm topology defines the flow of data between spouts that read data and bolts that process data. Storm guarantees processing of all data through its reliability APIs and guarantees no data loss even during failures.

Best practices for ansibleGeorge Shuklin

This document provides best practices for using Ansible including: - Break projects into common basics, specific configurations, non-standard software, and ad-hoc scripts. - Reduce scope so each piece works within its own domain without touching unrelated areas. - Name things properly to increase understandability. - Use Ansible lint to catch errors and improve code quality. - Use shell and command modules carefully and ensure idempotency or ability to detect changes. - Maintain staging environments that closely mimic production to test changes.

MongoDB: Scaling write performance | Devon 2012Daum DNA

This document discusses strategies for scaling write performance in MongoDB. It begins by noting that while MongoDB is easy to install and use for simple data models, scaling write performance becomes challenging, especially with secondary indexes. Maintaining high insert throughput requires careful consideration of the shard key to ensure even data distribution and avoid overloading indexes. Techniques discussed include partitioning collections by time, upgrading hardware with more RAM and IOPS, and choosing a low-cardinality hash prefix plus sequential field as the shard key. The key lessons are to understand how indexes impact performance, test shard key choices thoroughly, and recognize that linear scalability requires significant hardware resources.

High-Performance Storage Services with HailDB and Javasunnygleason

This document summarizes an approach to providing high-performance storage services using Java and HailDB. It discusses using the optimized "guts" of MySQL without needing to go through JDBC and SQL. It presents HailDB as a storage engine alternative to NoSQL options like Voldemort. It describes integrating HailDB with Java using JNA, building a REST API on top called St8, and examples of nifty applications like graph stores and counters. It concludes with discussing future work like improving packaging, online backup, and exploring JNI bindings.

More Related Content

What's hot (20)

Kafka replication apachecon_2013Jun Rao

Cassandra Introduction & FeaturesDataStax Academy

MongoDB Administration 101MongoDB

Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné

MySQL_SQL_Tunning_v0.1.3.docxNeoClova

RocksDB detailMIJIN AN

Introduction to MongoDBMike Dirolf

Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive

RocksDB compactionMIJIN AN

MongoDB WiredTiger Internals: Journey To TransactionsMydbops

mongodb와 mysql의 CRUD 연산의 성능 비교Woo Yeong Choi

How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015PostgreSQL-Consulting

MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...Jean-François Gagné

Redisimalik8088

InnoDB Locking Explained with Stick FiguresKarwin Software Solutions LLC

Parallel Replication in MySQL and MariaDBMydbops

A Technical Introduction to WiredTigerMongoDB

InnoDB Performance OptimisationMydbops

Introduction to Storm Chandler Huang

Best practices for ansibleGeorge Shuklin

Kafka replication apachecon_2013Jun Rao

Cassandra Introduction & FeaturesDataStax Academy

MongoDB Administration 101MongoDB

Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné

MySQL_SQL_Tunning_v0.1.3.docxNeoClova

RocksDB detailMIJIN AN

Introduction to MongoDBMike Dirolf

Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive

RocksDB compactionMIJIN AN

MongoDB WiredTiger Internals: Journey To TransactionsMydbops

mongodb와 mysql의 CRUD 연산의 성능 비교Woo Yeong Choi

How does PostgreSQL work with disks: a DBA's checklist in detail. PGConf.US 2015PostgreSQL-Consulting

MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...Jean-François Gagné

Redisimalik8088

InnoDB Locking Explained with Stick FiguresKarwin Software Solutions LLC

Parallel Replication in MySQL and MariaDBMydbops

A Technical Introduction to WiredTigerMongoDB

InnoDB Performance OptimisationMydbops

Introduction to Storm Chandler Huang

Best practices for ansibleGeorge Shuklin

Similar to Mongodb - Scaling write performance (20)

MongoDB: Scaling write performance | Devon 2012Daum DNA

High-Performance Storage Services with HailDB and Javasunnygleason

Introduction to NoSQLYan Cui

Scaling HDFS to Manage Billions of Files with Key-Value StoresDataWorks Summit

The document discusses scaling HDFS to manage billions of files. It describes how HDFS usage has grown from millions of files in 2007 to potentially billions of files in the future. To address this, the speakers propose storing HDFS metadata in a key-value store like LevelDB instead of solely in memory. They evaluate this approach and find comparable performance to HDFS for most operations. Future work includes improving operations like compaction and failure recovery in the new architecture.

Scaling HDFS to Manage Billions of FilesHaohui Mai

In this talk, we share our experience on designing and implementing a next-generation, scale-out architecture for HDFS. Particularly, we implement the namespace on top of a key-value store. Key-value stores are highly scalable and can be scaled out on demand. Our current prototype shows that under the new architecture the namespace of HDFS scales 10x better than the current generation with no performance loss, demonstrating that HDFS is capable of storing billions of files using the current hardware.

What every developer should know about database scalability, PyCon 2010jbellis

The document discusses the challenges of scaling databases and provides strategies for improving scalability. It covers: 1) The two main types of database operations that need to be scaled are reads and writes. Common "band-aid" approaches that don't actually scale the database are discussed. 2) Scaling reads can be improved through caching, partitioning caches, and replication. Scaling writes is more difficult and requires approaches like write-back caching, asynchronous replication, and data partitioning. 3) Partitioning data through sharding or vertical partitioning distributes the load across multiple database nodes. Consistent hashing is presented as a method to determine which server owns each data key. The challenges of non-transparent partitioning approaches are also

Accelerating NoSQLsunnygleason

This document summarizes Sunny Gleason's presentation on accelerating the NoSQL key-value store Voldemort by running it on the HailDB storage engine. It describes how Voldemort and HailDB work, the experimental setup comparing the performance of Voldemort using BDB-JE, Krati and HailDB for storage, and the results showing that HailDB provided the best performance. It concludes with ideas for future work such as improving the HailDB integration and exploring online backups.

Spring one2gx2010 spring-nonrelational_dataRoger Xia

This document provides a summary of a talk on using Spring with NoSQL databases. The talk discusses the benefits and drawbacks of NoSQL databases, and how the Spring Data project simplifies development of NoSQL applications. It then provides background on the two speakers, Chris Richardson and Mark Pollack. The agenda outlines explaining why NoSQL, overviewing some NoSQL databases, discussing Spring NoSQL projects, and having demos and code examples.

Hadoop Summit 2012 | Bayesian Counters AKA In Memory Data Mining for Large Da...Cloudera, Inc.

Processing of large data requires new approaches to data mining: low, close to linear, complexity and stream processing. While in the traditional data mining the practitioner is usually presented with a static dataset, which might have just a timestamp attached to it, to infer a model for predicting future/takeout observations, in stream processing the problem is often posed as extracting as much information as possible on the current data to convert them to an actionable model within a limited time window. In this talk I present an approach based on HBase counters for mining over streams of data, which allows for massively distributed processing and data mining. I will consider overall design goals as well as HBase schema design dilemmas to speed up knowledge extraction process. I will also demo efficient implementations of Naive Bayes, Nearest Neighbor and Bayesian Learning on top of Bayesian Counters.

Scaling with MongoDBMongoDB

This document summarizes strategies for scaling MongoDB databases. It discusses scaling through optimization techniques like schema design, indexing, and hardware configuration. It also covers horizontal scaling by adding shards. Specific topics covered include choosing a shard key, embedding data vs references, index design, and hardware considerations like disk and network performance. Replica sets and sharding architectures are presented as solutions for scaling reads and writes.

NickKallen_DataArchitectureAtTwitterScaleKostas Mavridis

This document discusses how Twitter handles big data in real-time. It describes three key data problems: tweets, timelines, and social graphs. For tweets, the data was originally stored relationally but now uses partitioning by time and user ID to improve performance. Timelines were initially slow to retrieve but are now pre-computed and stored in memory. The social graph was challenging due to its unbounded nature; it is now partitioned and stored with edges in both directions. Principles for handling real-time big data at scale include partitioning, indexing, replication, and keeping working data in memory.

Introduce Apache Cassandra - JavaTwo Taiwan, 2012Boris Yen

Cassandra is a distributed database that is highly scalable and fault tolerant. It uses a dynamic partitioning approach to distribute and replicate data across nodes. Cassandra offers tunable consistency levels and supports various client libraries like Hector and CQL for Java applications to interface with Cassandra. Some key features include horizontal scaling by adding nodes, replication of data for fault tolerance, and tunable consistency levels for reads and writes.

A Global In-memory Data System for MySQLDaniel Austin

Yes sql08 inmemorydbDaniel Austin

This document summarizes a presentation about using MySQL and the NDB storage engine to build a globally distributed in-memory database system on AWS. It proposes using MySQL/NDB clusters tiled across AWS availability zones to provide high availability and performance at a large scale. Key challenges discussed include managing data consistency across wide geographical distances and dealing with limitations of AWS like network performance and lack of global load balancing. Lessons learned are that NDB can successfully compete with NoSQL for most use cases by providing ACID compliance without sacrificing availability or performance.

Bayesian CountersDataWorks Summit

Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Ontico

Write-optimized database algorithms have been available in NoSQL products for many years. With MyRocks, the RocksDB storage engine for MySQL, we are using a write-optimized algorithm for a SQL DBMS. This talk will explain why we created MyRocks and how to compare write-optimized algorithms with the ubiquitous B-Tree in terms of read, write and space efficiency. A large MySQL deployment at Facebook is in the process of migrating from InnoDB to MyRocks. With RocksDB storage engines for MySQL and MongoDB and the Vinyl storage engine for Tarantool we think it is likely that you can consider a write-optimized database engine in the next few years.

What Every Developer Should Know About Database Scalabilityjbellis

Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will explain the advantages and limits of the approaches to scaling traditional relational databases, as well as the tradeoffs made by the designers of newer distributed systems like Cassandra. These slides are from Jonathan Ellis's OSCON 09 talk: http://en.oreilly.com/oscon2009/public/schedule/detail/7955

Mongodb labBas van Oudenaarde

Cassandra Summit 2014: Deploying Cassandra for Call of DutyDataStax Academy

Presenters: Seán O Sullivan, Service Reliability Engineer & Tim Czerniak, Software Engineer at Demonware This presentation covers the eight-month evaluation process we underwent to migrate some of Call of Duty’s core services from MySQL to Cassandra. We will outline our requirements, the process we followed for the evaluation, decisions we made around our schema, configuration and hardware, and some issues we encountered.

PhegData X - High Performance EBSHanson Dong

MongoDB: Scaling write performance | Devon 2012Daum DNA

High-Performance Storage Services with HailDB and Javasunnygleason

Introduction to NoSQLYan Cui

Scaling HDFS to Manage Billions of Files with Key-Value StoresDataWorks Summit

Scaling HDFS to Manage Billions of FilesHaohui Mai

What every developer should know about database scalability, PyCon 2010jbellis

Accelerating NoSQLsunnygleason

Spring one2gx2010 spring-nonrelational_dataRoger Xia

Hadoop Summit 2012 | Bayesian Counters AKA In Memory Data Mining for Large Da...Cloudera, Inc.

Scaling with MongoDBMongoDB

NickKallen_DataArchitectureAtTwitterScaleKostas Mavridis

Introduce Apache Cassandra - JavaTwo Taiwan, 2012Boris Yen

A Global In-memory Data System for MySQLDaniel Austin

Yes sql08 inmemorydbDaniel Austin

Bayesian CountersDataWorks Summit

Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Ontico

What Every Developer Should Know About Database Scalabilityjbellis

Mongodb labBas van Oudenaarde

Cassandra Summit 2014: Deploying Cassandra for Call of DutyDataStax Academy

PhegData X - High Performance EBSHanson Dong

More from Daum DNA (20)

Daum의 개방형 기술 전략 및 자바 기술 로드맵(2007)Daum DNA

Daum OAuth 2.0Daum DNA

Daum 음성인식 API (김한샘)Daum DNA

Daum 검색/지도 API (이정주)Daum DNA

오픈 API 활용방법(Daum 사례 중심, 윤석찬)Daum DNA

Daum 티스토리 API (천정환)Daum DNA

Daum 로그인 API (함태윤)Daum DNA

FT직군의 현재와 미래 - 홍윤표Daum DNA

웹접근성과 장애인 차별 금지법 - 장성민Daum DNA

반응형 웹 디자인은 만능인가? - 신현석Daum DNA

Daum devday 13 [bap]Daum DNA

Daum DevDay 13-힐링이 필요해Daum DNA

Daum DevDay 13 - 마음의 소리Daum DNA

Daum DevDay 13 - OpenBraceDaum DNA

Daum DevDay 13 - OgangjangDaum DNA

Daum DevDay 13 - MookDaum DNA

Daum DevDay 13 - MoonlightDaum DNA

Daum DevDay 13 - In-N-OutDaum DNA

Daum DevDay 13 - i-DFDaum DNA

Daum 키노트 | Devon 2012Daum DNA

Daum의 개방형 기술 전략 및 자바 기술 로드맵(2007)Daum DNA

Daum OAuth 2.0Daum DNA

Daum 음성인식 API (김한샘)Daum DNA

Daum 검색/지도 API (이정주)Daum DNA

오픈 API 활용방법(Daum 사례 중심, 윤석찬)Daum DNA

Daum 티스토리 API (천정환)Daum DNA

Daum 로그인 API (함태윤)Daum DNA

FT직군의 현재와 미래 - 홍윤표Daum DNA

웹접근성과 장애인 차별 금지법 - 장성민Daum DNA

반응형 웹 디자인은 만능인가? - 신현석Daum DNA

Daum devday 13 [bap]Daum DNA

Daum DevDay 13-힐링이 필요해Daum DNA

Daum DevDay 13 - 마음의 소리Daum DNA

Daum DevDay 13 - OpenBraceDaum DNA

Daum DevDay 13 - OgangjangDaum DNA

Daum DevDay 13 - MookDaum DNA

Daum DevDay 13 - MoonlightDaum DNA

Daum DevDay 13 - In-N-OutDaum DNA

Daum DevDay 13 - i-DFDaum DNA

Daum 키노트 | Devon 2012Daum DNA

Mongodb - Scaling write performance

1. MongoDB: Scaling write performance Junegunn Choi

2. MongoDB • Document data store • JSON-like document • Secondary indexes • Automatic failover • Automatic sharding

3. First impression: Easy • Easy installation • Easy data model • No prior schema design • Native support for secondary indexes

4. Second thought: Not so easy • No SQL • Coping with massive data growth • Setting up and operating sharded cluster • Scaling write performance

5. Today we’ll talk about insert performance

6. Insert throughput on a replica set

7. Steady 5k inserts/sec * 1kB record. ObjectId as PK * WriteConcern: Journal sync on Majority

8. Insert throughput on a replica set with a secondary index

10. Culprit: B+Tree index • Good at sequential insert • e.g. ObjectId, Sequence #, Timestamp • Poor at random insert • Indexes on randomly-distributed data

11. Sequential vs. Random insert 1 55 2 75 3 78 4 1 5 99 6 36 7 80 8 91 9 52 10 B+Tree 63 B+Tree 11 56 12 33 working set working set Sequential insert ➔ Small working set Random insert ➔ Large working set ➔ Fits in RAM ➔ Sequential I/O ➔ Cannot ﬁt in RAM ➔ Random I/O (bandwidth-bound) (IOPS-bound)

12. So, what do we do now?

13. 1. Partitioning Aug 2012 Sep 2012 Oct 2012 B+Tree ﬁts in memory does not ﬁt in memory

14. 1. Partitioning • MongoDB doesn’t support partitioning • Partitioning at application-level • e.g. Daily log collection • logs_20121012

15. Switch collection every hour

16. 2. Better H/W • More RAM • More IOPS • RAID striping • SSD • AWS Provisioned IOPS (1k ~ 10k)

18. 3. More H/W: Sharding • Automatic partitioning across nodes SHARD1 SHARD2 SHARD3 mongos router

19. 3 shards (3x3)

20. 3 shards (3x3) on RAID 1+0

21. There’s no free lunch • Manual partitioning • Incidental complexity • Better H/W • $ • Sharding • $$ • Operational complexity

22. “Do you really need that index?”

23. Scaling insert performance with sharding

24. = Choosing the right shard key

25. Shard key example: year_of_birth 64MB chunk ~ 1950 1971 ~ 1990 1951 ~ 1970 1991 ~ 2005 2006 ~ 2010 2010 ~ ∞ USERS USERS USERS SHARD1 SHARD2 SHARD3 mongos router

26. 5k inserts/sec w/o sharding

27. Sequential key • ObjectId as shard key • Sequence # • Timestamp

28. Worse throughput with 3x H/W.

29. Sequential key 1000 ~ 2000 • All inserts into one chunk 5000 ~ 7500 • Cannot scale insert performance 9000 ~ ∞ • Chunk migration overhead USERS SHARD-x 9001, 9002, 9003, 9004, ...

30. Sequential key

31. Hash key • e.g. SHA1(_id) = 9f2feb0f1ef425b292f2f94 ... • Distributes inserts evenly across all chunks

33. Hash key • Performance drops as collection grows • Why? Mandatory index on shard key • B+Tree problem again!

34. Sequential key Hash key

35. Sequential + hash key • Coarse-grained sequential preﬁx • e.g. Year-month + hash value • 201210_24c3a5b9 B+Tree 201208_* 201209_* 201210_*

36. But what if... B+Tree large working set 201208_* 201209_* 201210_*

37. Sequential + hash key • Can you predict data growth rate? • Balancer not clever enough • Only considers # of chunks • Migration slow during heavy-writes

38. Sequential key Hash key Sequential + hash key

39. Low-cardinality hash key • Small portion of hash value Shard key range: A ~ D • e.g. A~Z, 00~FF • Alleviates B+Tree problem Local • Sequential access on ﬁxed # B+Tree of parts • Cardinality / # of shards A A A B B B C C C

41. Low-cardinality hash key • Limits the # of possible chunks • e.g. 00 ~ FF ➔ 256 chunks • Chunk grows past 64MB • Balancing becomes difﬁcult

42. Sequential key Hash key Sequential + hash key Low-cardinality hash key

43. Low-cardinality hash preﬁx + sequential part Shard key range: A000 ~ C999 • e.g. Short hash preﬁx + timestamp • Nice index access pattern Local B+Tree • Unlimited number of chunks A000 A123 B000 B123 C000 C123

44. Finally, 2x throughput

45. Lessons learned • Know the performance impact of secondary index • Choose the right shard key • Test with large data sets • Linear scalability is hard • If you really need it, consider HBase or Cassandra • SSD

46. Thank you. Questions? 유응섭 [email protected] 최준건 [email protected]

Mongodb - Scaling write performance

Recommended

More Related Content

What's hot (20)

Similar to Mongodb - Scaling write performance (20)

More from Daum DNA (20)

Mongodb - Scaling write performance