NoSQL Infrastructure

May 20, 20130 likes834 views

This document discusses MongoDB infrastructure at Server Density. It notes that Server Density uses 27 MongoDB nodes to store 20TB of data per month from their MySQL database. Some key reasons for choosing MongoDB include replication, official drivers, easy deployment, and fast performance out of the box. The document then discusses various MongoDB performance and infrastructure considerations like network throughput, replication lag, failover processes, disk types, backups, and monitoring.

~3333 write ops/s 0.07 - 0.05 ms response

• June 2009 - 4yrs
•27 nodes
MongoDB at Server Density

•MySQL -> MongoDB
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

•MySQL -> MongoDB
•20TB data per month
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

Why?
• Replication
• Ofﬁcial drivers
• Easy deployment

Why?
• Replication
• Ofﬁcial drivers
• Easy deployment
• Fast out of the box

Picture is unrelated! Mmm, ice cream.
• Fast network
Performance

• Fast network
Performance
EC2 10 Gigabit Ethernet
- Cluster Compute
- High Memory Cluster
- Cluster GPU
- High I/O
- High Storage

• Fast network
Performance
Workload: Read/Write?
Result set size
What is being stored?

• Fast network
Performance
Use Network Throughput
Normal 0-100Mb/s
Replication (Initial Sync) Burst +100Mb/s
Replication (Oplog) 0-100Mb/s
Backup Initial Sync + Oplog

• Fast network
Performance
Inter-DC LAN
Cross USA Washington, DC - San Jose, CA

• Fast network
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Paciﬁc 150ms
Europe - Japan 300ms

Failover
•Replica sets
•Min 3 nodes
•Master/slave

Failover
•Replica sets
•Min 3 nodes
•Master/slave
•Automatic failover

• Replication lag
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Paciﬁc 150ms
Europe - Japan 300ms

Replication Lag
1. Reads: eventual consistency

Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind

Eventual Consistency
Stale data
Inconsistent data

Eventual Consistency
Stale data
Inconsistent data
Changing data

Eventual Consistency
Use Case Needs consistency?
Graphs No
User proﬁle Yes
Statistics Depends
Alert conﬁg Yes

Slave behind
Failover: out of date master

MongoDB WriteConcern
Changed Nov 27 2012

• Safe by default
>>> from pymongo import MongoClient
>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Primary
2 Primary + x1 secondary
3 Primary + x2 secondaries
MongoDB WriteConcern

$Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})$

${ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags$

Picture is unrelated! Mmm, ice cream.
• Fast network
•More RAM
Performance

http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

More RAM = expensive
Performance
x2 4GB RAM 12 month Prices

EC2 disk/RAM pricing
Performance
$2232/m
$2520/m
$43/m
$295/m

Tips: rand()
•Field names
•Covered indexes

Tips: rand()
•Field names
•Covered indexes
•Collections / databases

Backups
What is the use case?
Fixing user errors?
Point in time restore?
Disaster recovery?

Backups
•Disaster recovery
Age
Offsite
Restore time

Backups
Frequency
Consistency
Veriﬁcation

www.ﬂickr.com/photos/daddo83/3406962115/
Monitoring
•System
Disk i/o
Disk use

www.ﬂickr.com/photos/daddo83/3406962115/
Monitoring
Disk i/o
Disk use
•System
Swap

www.ﬂickr.com/photos/daddo83/3406962115/
Monitoring
Optime
State
•Replication

Monitoring tools
www.serverdensity.com/nosql

David Mytton
david@serverdensity.com
@davidmytton
Woop Japan!
www.serverdensity.com/nosql

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment. This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

Handling 20 billion requests a monthDmitriy Dumanskiy

This document discusses the architecture and technical challenges of handling a large volume of requests for an online advertising platform. It summarizes three key projects handled by the platform that delivered 3 billion, 14 billion, and 20 billion requests per month respectively. It describes the technologies used, including Solr, Redis, MySQL, Hadoop and Amazon Web Services instances. It also outlines optimizations made to improve performance, such as data compression, query optimizations, and Java 7 improvements. The goal was to process over 11,000 requests per second on average while maintaining response times below 100ms.

Tweaking performance on high-load projectsDmitriy Dumanskiy

This document discusses optimizing performance for high-load projects. It summarizes the delivery loads and technologies used for several projects including mGage, mobclix and XXXX. It then discusses optimizations made to improve performance, including using Solr for search, Redis for real-time data, Hadoop for reporting, and various Java optimizations in moving to Java 7. Specific optimizations discussed include reducing garbage collection, improving random number generation, and minimizing I/O operations.

JEEConf. Vanilla javaDmitriy Dumanskiy

This document discusses various approaches to building high performance Java applications for handling IoT workloads. It begins by noting that vanilla Java can handle 10,000 requests per second per CPU core. It then discusses blocking and non-blocking IO approaches, highlighting several non-blocking frameworks like Netty. The document also summarizes the architecture and performance of the Blynk IoT platform, which uses a non-blocking architecture with batches, no synchronization, and in-memory structures to achieve high throughput. Finally, it outlines different versions of the Blynk platform with increasing scale and resilience.

Back to Basics Webinar 3: Introduction to Replica SetsMongoDB

This document provides an introduction to MongoDB replica sets, which allow for data redundancy and high availability. It discusses how replica sets work, including the replica set life cycle and how applications should handle writes and queries when using a replica set. Specifically, it explains that the MongoDB driver is responsible for server discovery and monitoring, retry logic, and handling topology changes in a replica set to provide a consistent view of the data to applications.

JVM performance options. How it worksDmitriy Dumanskiy

Performance Tuning and OptimizationMongoDB

This document provides an overview of performance tuning and optimization in MongoDB. It defines performance tuning as modifying a system to handle increased load, while optimization is modifying a system to work more efficiently or use fewer resources. Measurement tools discussed include log files, the profiler, query optimizer, and explain plans. Effecting change involves measuring current performance, identifying bottlenecks, removing bottlenecks, remeasuring, and repeating. Possible areas for improvement discussed are schema design, access patterns, indexing, hardware configuration, and instance configuration. The document provides examples and best practices around indexing, access patterns, and hardware tuning.

Back to Basics: My First MongoDB ApplicationMongoDB

- The document is a slide deck for a webinar on building a basic blogging application using MongoDB. - It covers MongoDB concepts like documents, collections and indexes. It then demonstrates how to install MongoDB, connect to it using the mongo shell, and insert documents. - The slide deck proceeds to model a basic blogging application using MongoDB, creating collections for users, articles and comments. It shows how to query, update, and import large amounts of seeded data.

Latinowarekchodorow

The document discusses MongoDB, a non-relational database that uses documents with a flexible schema rather than tables, and is well-suited for applications that need to store data in complex, nested structures. It provides an overview of key MongoDB concepts like collections, queries, indexing, updating documents, and aggregation capabilities. Examples are given of how MongoDB can be used for applications involving user profiles, blogs, and logging.

MongoDB - Ekino PHPFlorent DENIS

MongoDB-SESSION03Jainul Musani

Introduction to MongoDBNosh Petigara

MongoDB Shell Tips & TricksMongoDB

The document discusses tips and tricks for using the MongoDB shell. It describes what the shell is, advantages like debugging queries and administration, and disadvantages like numbers and dates. It provides examples of shell commands and functions for tasks like loading scripts, running commands, profiling, and administration. It also discusses configuration options and ways to improve the shell experience like keyboard shortcuts and the .mongorc.js file.

mongoDB PerformanceMoshe Kaplan

The document discusses MongoDB performance optimization strategies presented by Moshe Kaplan at a VP R&D Open Seminar. It covers topics like sharding, in-memory databases, MapReduce, profiling, indexes, server stats, schema design, and locking in MongoDB. Slides include information on tuning configuration parameters, analyzing profiling results, explain plans, index management, and database stats.

MongoDB: Comparing WiredTiger In-Memory Engine to RedisJason Terpko

Back to Basics, webinar 2: La tua prima applicazione MongoDBMongoDB

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

MongoDB is easy to download and run locally but requires some thought and further understanding when deploying to production. At scale, schema design, indexes and query patterns really matter. So does data structure on disk, sharding, replication and data centre awareness. This talk will examine these factors in the context of analytics, and more generally, to help you optimise MongoDB for any scale. Presented at MongoDB Days London 2013 by David Mytton.

Back to Basics Webinar 5: Introduction to the Aggregation FrameworkMongoDB

The document provides information about an upcoming webinar on the MongoDB aggregation framework. Key details include: - The webinar will introduce the aggregation framework and provide an overview of its capabilities for analytics. - Examples will use a real-world vehicle testing dataset to demonstrate aggregation pipeline stages like $match, $project, and $group. - Attendees will learn how the aggregation framework provides a simpler way to perform analytics compared to other tools like Spark and Hadoop.

Back to Basics Webinar 1: Introduction to NoSQLMongoDB

This document provides an overview of an introduction to NoSQL webinar. It discusses why NoSQL databases were created, the different types of NoSQL databases including key-value stores, column stores, graph stores, multi-model databases and document stores. It provides details on MongoDB, describing how MongoDB stores data as JSON-like documents with dynamic schemas and supports features like indexing, aggregation and geospatial queries. The webinar agenda is also outlined.

Getting Started with MongoDB and NodeJSMongoDB

For developers new to MongoDB and Node.js, however, some the common design patterns are very different than those of a RDBMS and traditional synchronous languages. Developers learning these technologies together may find it a bit bewildering. In reality, however, these tools fit perfectly together and enable I high degree of developer productivity and application performance. This webinar will walk developers through common MongoDB development patterns in Node.js, such as efficiently loading data into MongoDB using MongoDB's bulk API, iterating through query results, and managing simultaneous asynchronous MongoDB queries to provide the best possible application performance. Working Node.js and MongoDB examples will be used throughout the presentation.

MongoDB : The Definitive GuideWildan Maulana

This document is a chapter-by-chapter summary of the book "MongoDB: The Definitive Guide". It covers topics such as getting started with MongoDB, creating, updating and deleting documents, querying, indexing, aggregation, administration, replication and sharding. Example applications discussed include a chemical search engine built with Java, a news aggregator using PHP, a custom form submission application in Ruby, and a real-time analytics application in Python. Additional appendices provide information on installing MongoDB and details about the mongo shell and MongoDB internals.

Python and MongoDB Norberto Leite

MongoDB - External AuthenticationJason Terpko

This presentation will discuss implementing external authentication when using Percona Server for MongoDB and MongoDB Enterprise. It will review authentication using OpenLDAP or ActiveDirectory and ActiveDirectory with Kerberos. The presentation will also include examples of the configurations required by these external directory services. It will also review the LDAP Authorization features introduced in MongoDB Enterprise 3.4.

MongoDB and PythonNorberto Leite

MongoDB is one of the most popular databases these days and there are a few reasons for such popularity. One of these reasons is the excellent integration with different programming languages and development frameworks. In the case of Python we take it a few notches up (native use of dictionaries, integration with asynchronous libraries (twisted, gevent), good support for web frameworks like django, flask, bottle ... (mongoengine anyone?). This talk is about the several different projects that we support, the way to effectively use Python and MongoDB together and a few other improvements and announcements.

High Performance Applications with MongoDBMongoDB

MongoDBSteven Francia

Back to Basics Webinar 2: Your First MongoDB ApplicationMongoDB

MongoDB - Sharded Cluster TutorialJason Terpko

This document provides an overview and instructions for deploying, upgrading, and troubleshooting a MongoDB sharded cluster. It describes the components of a sharded cluster including shards, config servers, and mongos processes. It provides recommendations for initial deployment including using replica sets for shards and config servers, DNS names instead of IPs, and proper user authorization. The document also outlines best practices for upgrading between minor and major versions, including stopping the balancer, upgrading processes in rolling fashion, and handling incompatible changes when downgrading major versions.

Navigating the customer lifecycleServer Density

Puppet Camp Ghent 2013Server Density

This document discusses adding Forge modules to Puppet Enterprise. It describes moving the HTTP load balancer from Pound to nginx, keeping existing manifests pulled from GitHub, and using the Puppet Forge to install the puppetlabs/nginx module or integrating it via Git submodules. It also covers parameterizing classes on the Puppet Enterprise console and merging site.pp files, as well as updating nginx configurations on the fly and considering alternative nginx modules on the Forge.

More Related Content

What's hot (20)

Latinowarekchodorow

MongoDB - Ekino PHPFlorent DENIS

MongoDB-SESSION03Jainul Musani

Introduction to MongoDBNosh Petigara

MongoDB Shell Tips & TricksMongoDB

mongoDB PerformanceMoshe Kaplan

MongoDB: Comparing WiredTiger In-Memory Engine to RedisJason Terpko

Back to Basics, webinar 2: La tua prima applicazione MongoDBMongoDB

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

Back to Basics Webinar 5: Introduction to the Aggregation FrameworkMongoDB

Back to Basics Webinar 1: Introduction to NoSQLMongoDB

Getting Started with MongoDB and NodeJSMongoDB

MongoDB : The Definitive GuideWildan Maulana

Python and MongoDB Norberto Leite

MongoDB - External AuthenticationJason Terpko

MongoDB and PythonNorberto Leite

High Performance Applications with MongoDBMongoDB

MongoDBSteven Francia

Back to Basics Webinar 2: Your First MongoDB ApplicationMongoDB

MongoDB - Sharded Cluster TutorialJason Terpko

Latinowarekchodorow

MongoDB - Ekino PHPFlorent DENIS

MongoDB-SESSION03Jainul Musani

Introduction to MongoDBNosh Petigara

MongoDB Shell Tips & TricksMongoDB

mongoDB PerformanceMoshe Kaplan

MongoDB: Comparing WiredTiger In-Memory Engine to RedisJason Terpko

Back to Basics, webinar 2: La tua prima applicazione MongoDBMongoDB

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

Back to Basics Webinar 5: Introduction to the Aggregation FrameworkMongoDB

Back to Basics Webinar 1: Introduction to NoSQLMongoDB

Getting Started with MongoDB and NodeJSMongoDB

MongoDB : The Definitive GuideWildan Maulana

Python and MongoDB Norberto Leite

MongoDB - External AuthenticationJason Terpko

MongoDB and PythonNorberto Leite

High Performance Applications with MongoDBMongoDB

MongoDBSteven Francia

Back to Basics Webinar 2: Your First MongoDB ApplicationMongoDB

MongoDB - Sharded Cluster TutorialJason Terpko

Viewers also liked (19)

Navigating the customer lifecycleServer Density

Puppet Camp Ghent 2013Server Density

Handling incidentsServer Density

The document discusses how to handle incidents, downtime, and outages. It notes that the cost of downtime for companies in Q1 2015 was $2.9 billion, $870 million, and $4.1 billion. It recommends preparing for incidents by having on-call staff and documentation, responding quickly by following an incident response checklist and notifying stakeholders, and performing a postmortem within days to analyze what failed and how to prevent future issues.

StartOps: Growing an ops team from 1 founderServer Density

Bootstrapped startups don't have the luxury of a full team of ops engineers available to respond to issues 24/7, so how can you survive on your own? This talk will tell the story of how to run your infrastructure as a single founder through to growing that into a team of on call engineers. It will include some interesting war stories as well as tips and suggestions for how to run ops at a startup. Presented at DevOpsDays London 2013 by David Mytton.

Experiences from DevOps production: Deployment, performance, failure.Server Density

Puppet Camp 2012Server Density

Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density. Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed. Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances. www.serverdensity.com/puppetcamp/

High performance Infrastructure Oct 2013Server Density

The document discusses high performance infrastructure for Server Density which includes 150 servers that have been running since June 2009 and migrated from MySQL to MongoDB. It stores 25TB of data per month. Key aspects of performance discussed are using fast networks like 10 Gigabit Ethernet on AWS, ensuring high memory, using SSDs over spinning disks for performance, and factors like replication lag based on location. The document also compares options like using cloud, dedicated servers, or colocation and discusses monitoring, backups, dealing with outages, and other operational aspects.

Briefing: ContainersServer Density

Containers seem to have suddenly become the hot new thing everyone is talking about, but what are they? Why are they important? How should you use them and what does it mean for cloud infrastructure? This talk will examine the history, technical details and strategy around containerisation from the perspective of developers and operations, consider internal container OSs like Rocket and Ubuntu Core as well as management layers like Docker and Apache Mesos and take a look at why cloud providers are launching their own services around them. Presented by David Mytton at Datacloud Monaco 2015-06-04

Remote startup - building a company from everywhere in the worldServer Density

This document discusses how Server Density grew from 2 employees in 2009 to 12 employees in 2013 while being fully remote. It outlines the company's timeline and some advantages of being fully remote such as access to worldwide talent, lower costs, and no distractions of an office. However, it also notes disadvantages like collaboration being more difficult without in-person interactions. While the company added an office in 2012, the document emphasizes that working remotely is a mindset that is difficult to adopt after initially co-locating. Effective communication and availability are important for fully distributed teams.

Scaling humans - Ops teams and incident managementServer Density

The document discusses the costs of downtime for companies and best practices for incident management teams to prepare for, respond to, and review incidents to minimize downtime costs. It notes that downtime costs companies billions per quarter and recommends teams prepare documentation and contact information, have on-call rotation schedules, log all responses to incidents, provide frequent status updates, gather teams to escalate incidents as needed, and conduct post-mortem reviews within days of an incident.

DevOps Incident Handling - Making friends not enemies.Server Density

Why puppet? Why now?Server Density

Why Puppet? Why now? Can you get by without using any config management? You probably think don't have time, or that your project is too small. What can using Puppet really add? How can you justify investing time up front? Maybe you can just do it later? Getting started with config management can often seem like a big project, especially if you only manage a few systems or have a small team. This talk will examine why you should use Puppet from the beginning. It will examine what you can do with Puppet that couldn't do otherwise, how much time it will save and why it's especially important if you think your project has even the smallest chance of scaling in the future. Presented by David Mytton at Puppet Camp London 2015-04-13

Puppet at the centre of everythingServer Density

The document discusses Server Density's architecture which includes 100 Ubuntu servers with 50% being virtual, using Nginx, Python, and MongoDB. It handles 25TB of data per month. Puppet is used for configuration, failover, code deploys, and system updates. The document also considers colocating servers versus using a dedicated provider and factors like hardware specs, costs, skills required, and fun.

Infrastructure choices - cloud vs colo vs bare metalServer Density

This document discusses the differences between cloud, colocation, and bare metal infrastructure options. It covers key considerations for performance including CPU, memory, disk, and network latency and bandwidth. Colocation provides hardware at a specific location while maintaining internal skills, but has costs for total spend, hardware specifications, and power usage. Cloud infrastructure offers elastic workloads and support for demand spikes and unknown requirements, but bare metal is preferable for managed hardware replacement and networking needs. Overall, the best option depends on an organization's specific workload characteristics and skills.

Flight training for DevOps & HumanOps - IncontroDevOps 2016Server Density

Flight training for DevOpsServer Density

This document discusses improving incident response procedures through practices such as checklists, documented procedures, realistic incident simulations and postmortems. It recommends extended use of checklists to guide responses while still allowing for experience and independent thought. Regular incident response simulations that test both general processes and specific failures can help refine procedures and build confidence. Postmortems should objectively review incidents, suggest improvements and run through scenarios again over time to prevent complacency.

Content marketing @ Server DensityServer Density

How to monitor NGINXServer Density

How to Monitor MySQLServer Density

Navigating the customer lifecycleServer Density

Puppet Camp Ghent 2013Server Density

Handling incidentsServer Density

StartOps: Growing an ops team from 1 founderServer Density

Experiences from DevOps production: Deployment, performance, failure.Server Density

Puppet Camp 2012Server Density

High performance Infrastructure Oct 2013Server Density

Briefing: ContainersServer Density

Remote startup - building a company from everywhere in the worldServer Density

Scaling humans - Ops teams and incident managementServer Density

DevOps Incident Handling - Making friends not enemies.Server Density

Why puppet? Why now?Server Density

Puppet at the centre of everythingServer Density

Infrastructure choices - cloud vs colo vs bare metalServer Density

Flight training for DevOps & HumanOps - IncontroDevOps 2016Server Density

Flight training for DevOpsServer Density

Content marketing @ Server DensityServer Density

How to monitor NGINXServer Density

How to Monitor MySQLServer Density

Similar to NoSQL Infrastructure (20)

Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...MongoDB

MongoDB for Time Series Data Part 3: ShardingMongoDB

The document discusses sharding time series sensor data in MongoDB. It recommends modeling the application's read, write and storage patterns to determine the optimal sharding strategy. A good shard key has sufficient cardinality, distributes writes evenly and enables targeted reads. For time series data, a compound shard key of an arbitrary value and incrementing timestamp is suggested to balance hot spots and targeted queries. The document also covers configuring a sharded cluster and replica sets with tags to control data distribution.

OSDC 2012 | Scaling with MongoDB by Ross LawleyNETWAYS

MongoDB's architecture features built-in support for horizontal scalability, and high availability through replica sets. Auto-sharding allows users to easily distribute data across many nodes. Replica sets enable automatic failover and recovery of database nodes within or across data centers. This session will provide an introduction to scaling with MongoDB by one of the developers working on the project.

Maintenance for MongoDB Replica SetsIgor Donchovski

Redundancy and high availability are the basis for all production deployments. With MongoDB high availability is achieved with replica sets which provides automatic fail-over in case the Primary goes down. In this session we will review multiple maintenance scenarios that will include the proper steps for keeping the high availability while we perform the maintenance steps without causing downtime. This session will cover Database upgrades, OS server patching, Hardware upgrades, Network maintenance and more. How MongoDB HA works Replica sets components/deployment typologies Database upgrades System patching/upgrade Network maintenance Add/Remove members to the replica set Reconfiguring replica set members Building indexes Backups and restores

Webinar: Replication and Replica SetsMongoDB

For the first time this year, 10gen will be offering a track completely dedicated to Operations at MongoSV, 10gen's annual MongoDB user conference on December 4. Learn more at MongoSV.com MongoDB supports replication for failover and redundancy. In this session we will introduce the basic concepts around replica sets, which provide automated failover and recovery of nodes. We'll show you how to set up, configure, and initiate a replica set, and methods for using replication to scale reads. We'll also discuss proper architecture for durability.

Replication and Replica SetsMongoDB

2013 london advanced-replicationMarc Schwering

This document discusses advanced replication in MongoDB replica sets. It begins by explaining the roles and configuration of replica set members, including primaries, secondaries, and arbiters. It then covers the implementation details of replication using an oplog and heartbeats between nodes. The rest of the document discusses best practices for maintenance, data center awareness using tagging, write concerns, and read preferences to control data distribution and failover.

Mongo db roma replication and shardingGuglielmo Incisa Di Camerana

Replication and sharding allow MongoDB databases to scale horizontally across commodity servers for high availability and increased performance. Replication duplicates data across multiple nodes so the application can continue running if a node fails. Sharding partitions data across nodes to distribute load and storage requirements. MongoDB supports combining replication within shards for redundancy and sharding to partition data across replicated shards, providing redundancy and scalability.

Replication and Replica SetsMongoDB

This document discusses MongoDB replica sets, which provide redundancy and increase availability. It covers the replica set lifecycle including creation, initialization, failure, failover and recovery. Configuration options like priority, hidden status and slave delay are explained. Methods for developing with replica sets like write concern, tagging and read preference modes are also summarized. Finally, operational considerations for maintenance, upgrades and multi-data center deployments are briefly discussed.

Advanced ReplicationMongoDB

MongoDB London 2013: Basic Replication in MongoDB presented by Marc Schwering...MongoDB

The document discusses MongoDB replication and replica sets. It covers the lifecycle of replica sets including creation, initialization, failure, and recovery. It also discusses replica set roles and configuration options. Additionally, it addresses considerations for developing with replica sets like strong/delayed consistency, write concerns, tagging, and read preferences. Finally, it discusses operational considerations like maintenance/upgrades and different replica set deployment architectures.

Mongodb workshopHarun Yardımcı

This document discusses MongoDB replication and sharding. It begins with an introduction to MongoDB and its data model. It then covers replication with master-slave and replica sets. It also discusses sharding components including shard servers, config servers, and mongos routers. It provides examples of setting up replica sets, adding shards, and combining replica sets with sharding. The document encourages trying these techniques yourself and sharing experiences.

The powerful toolset of the go-mysql libraryDaniël van Eeden

Bio bigdata Mk Kim

This document provides an overview of bio big data and related technologies. It discusses what big data is and why bio big data is necessary given the large size of genomic data sets. It then outlines and describes Hadoop, Spark, machine learning, and streaming in the context of bio big data. For Hadoop, it explains HDFS, MapReduce, and the Hadoop ecosystem. For Spark, it covers RDDs, Spark SQL, MLlib, and Spark Streaming. The document is intended as an introduction to key concepts and tools for working with large biological data sets.

Eagle6 mongo dc revisedMongoDB

Eagle6 Enterprise Situational AwarenessMongoDB

Eagle6 is a product that use system artifacts to create a replica model that represents a near real-time view of system architecture. Eagle6 was built to collect system data (log files, application source code, etc.) and to link system behaviors in such a way that the user is able to quickly identify risks associated with unknown or unwanted behavioral events that may result in unknown impacts to seemingly unrelated down-stream systems. This session is designed to present the capabilities of the Eagle6 modeling product and how we are using MongoDB to support near-real-time analysis of large disparate datasets.

Webinar: Replication and Replica SetsMongoDB

Building a Scalable Distributed Stats Infrastructure with Storm and KairosDBCody Ray

Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB Many startups collect and display stats and other time-series data for their users. A supposedly-simple NoSQL option such as MongoDB is often chosen to get started... which soon becomes 50 distributed replica sets as volume increases. This talk describes how we designed a scalable distributed stats infrastructure from the ground up. KairosDB, a rewrite of OpenTSDB built on top of Cassandra, provides a solid foundation for storing time-series data. Unfortunately, though, it has some limitations: millisecond time granularity and lack of atomic upsert operations which make counting (critical to any stats infrastructure) a challenge. Additionally, running KairosDB atop Cassandra inside AWS brings its own set of challenges, such as managing Cassandra seeds and AWS security groups as you grow or shrink your Cassandra ring. In this deep-dive talk, we explore how we've used a mix of open-source and in-house tools to tackle these challenges and build a robust, scalable, distributed stats infrastructure.

Tweaking perfomance on high-load projects_Думанский ДмитрийGeeksLab Odessa

This document discusses optimizing the performance of several high-load projects delivering billions of requests per month. It summarizes the evolution and delivery loads of different projects over time. It then analyzes the technical stacks and architectures used, identifying problems and solutions implemented around areas like querying, data storage, processing, and networking. Key lessons learned are around sharding and resharding data, optimizing I/O, using streaming processing like Storm over batch processing like Hadoop, and working within AWS limits and capabilities.

Get expertise with mongo dbAmit Thakkar

Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...MongoDB

MongoDB for Time Series Data Part 3: ShardingMongoDB

OSDC 2012 | Scaling with MongoDB by Ross LawleyNETWAYS

Maintenance for MongoDB Replica SetsIgor Donchovski

Webinar: Replication and Replica SetsMongoDB

Replication and Replica SetsMongoDB

2013 london advanced-replicationMarc Schwering

Mongo db roma replication and shardingGuglielmo Incisa Di Camerana

Replication and Replica SetsMongoDB

Advanced ReplicationMongoDB

MongoDB London 2013: Basic Replication in MongoDB presented by Marc Schwering...MongoDB

Mongodb workshopHarun Yardımcı

The powerful toolset of the go-mysql libraryDaniël van Eeden

Bio bigdata Mk Kim

Eagle6 mongo dc revisedMongoDB

Eagle6 Enterprise Situational AwarenessMongoDB

Webinar: Replication and Replica SetsMongoDB

Building a Scalable Distributed Stats Infrastructure with Storm and KairosDBCody Ray

Tweaking perfomance on high-load projects_Думанский ДмитрийGeeksLab Odessa

Get expertise with mongo dbAmit Thakkar

Recently uploaded (20)

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Mobile App Development Company in Saudi ArabiaSteve Jonas

EmizenTech is a globally recognized software development company, proudly serving businesses since 2013. With over 11+ years of industry experience and a team of 200+ skilled professionals, we have successfully delivered 1200+ projects across various sectors. As a leading Mobile App Development Company In Saudi Arabia we offer end-to-end solutions for iOS, Android, and cross-platform applications. Our apps are known for their user-friendly interfaces, scalability, high performance, and strong security features. We tailor each mobile application to meet the unique needs of different industries, ensuring a seamless user experience. EmizenTech is committed to turning your vision into a powerful digital product that drives growth, innovation, and long-term success in the competitive mobile landscape of Saudi Arabia.

tecnologias de las primeras civilizaciones.pdffjgm517

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how: • Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules. • Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance. • Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity. • Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications. • Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market. With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications. Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Building 10x Organizations with Modern Productivity Metrics 10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’ Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Mobile App Development Company in Saudi ArabiaSteve Jonas

tecnologias de las primeras civilizaciones.pdffjgm517

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

NoSQL Infrastructure

1. NoSQL Infrastructure

2. ~3333 write ops/s 0.07 - 0.05 ms response

3. David Mytton Woop Japan!

5. MongoDB at Server Density

6. •27 nodes MongoDB at Server Density

7. • June 2009 - 4yrs •27 nodes MongoDB at Server Density

8. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - 4yrs

9. •MySQL -> MongoDB •20TB data per month •27 nodes MongoDB at Server Density • June 2009 - 4yrs

10. Why?

11. Why? • Replication

12. Why? • Replication • Ofﬁcial drivers

13. Why? • Replication • Ofﬁcial drivers • Easy deployment

14. Why? • Replication • Ofﬁcial drivers • Easy deployment • Fast out of the box

15. It’s a little different.

16. Picture is unrelated! Mmm, ice cream. • Fast network Performance

17. • Fast network Performance EC2 10 Gigabit Ethernet - Cluster Compute - High Memory Cluster - Cluster GPU - High I/O - High Storage

18. • Fast network Performance Workload: Read/Write? Result set size What is being stored?

19. • Fast network Performance Use Network Throughput Normal 0-100Mb/s Replication (Initial Sync) Burst +100Mb/s Replication (Oplog) 0-100Mb/s Backup Initial Sync + Oplog

20. • Fast network Performance Inter-DC LAN

21. • Fast network Performance Inter-DC LAN Cross USA Washington, DC - San Jose, CA

22. • Fast network Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Paciﬁc 150ms Europe - Japan 300ms

23. Failover •Replica sets

24. Failover •Replica sets •Master/slave

25. Failover •Replica sets •Min 3 nodes •Master/slave

26. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover

27. • Replication lag Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Paciﬁc 150ms Europe - Japan 300ms

28. Replication Lag 1. Reads: eventual consistency

29. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind

30. Eventual Consistency Stale data

31. Eventual Consistency Stale data Inconsistent data

32. Eventual Consistency Stale data Inconsistent data Changing data

33. Eventual Consistency Use Case Needs consistency? Graphs No User proﬁle Yes Statistics Depends Alert conﬁg Yes

34. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind

35. Slave behind Failover: out of date master

36. MongoDB WriteConcern Changed Nov 27 2012

37. • Safe by default >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value Meaning 0 Unsafe 1 Primary 2 Primary + x1 secondary 3 Primary + x2 secondaries MongoDB WriteConcern

39. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})

40. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags

41. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags

42. Picture is unrelated! Mmm, ice cream. • Fast network •More RAM Performance

43. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

44. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

45. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

46. More RAM = expensive Performance x2 4GB RAM 12 month Prices

47. RAM SSDs Spinning disk Cost Speed

48. Softlayer disk pricing Performance

49. EC2 disk/RAM pricing Performance $2232/m $2520/m $43/m $295/m

50. SSD vs Spinning Performance

51. SSD vs Spinning Performance

52. SSD vs Spinning Performance

53. Tips: rand() •Field names