NoSQL Infrastructure - Late 2013

Oct 9, 20130 likes1,721 views

NoSQL databases are often touted for their performance and whilst it's true that they usually offer great performance out of the box, it still really depends on how you deploy your infrastructure. Dedicated vs cloud? In memory vs on disk? Spindal vs SSD? Replication lag. Multi data centre deployment. This talk considers all the infrastructure requirements of a successful high performance infrastructure with hints and tips that can be applied to any NoSQL technology. It includes things like OS tweaks, disk benchmarks, replication, monitoring and backups.

~3333 write ops/s 0.07 - 0.05 ms response

• June 2009 - 4yrs
•27 nodes
MongoDB at Server Density

•MySQL -> MongoDB
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

•MySQL -> MongoDB
•25TB data per month
•27 nodes
MongoDB at Server Density
• June 2009 - 4yrs

Why?
• Replication
• Ofﬁcial drivers
• Easy deployment

Why?
• Replication
• Ofﬁcial drivers
• Easy deployment
• Fast out of the box

Picture is unrelated! Mmm, ice cream.
• Fast network
Performance

• Fast network
Performance
EC2 10 Gigabit Ethernet
- Cluster Compute
- High Memory Cluster
- Cluster GPU
- High I/O
- High Storage

• Fast network
Performance
Workload: Read/Write?
Result set size
What is being stored?

• Fast network
Performance
Use Network Throughput
Normal 0-100Mb/s
Replication (Initial Sync) Burst +100Mb/s
Replication (Oplog) 0-100Mb/s
Backup Initial Sync + Oplog

• Fast network
Performance
Inter-DC LAN
Cross USA Washington, DC - San Jose, CA

• Fast network
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Paciﬁc 150ms
Europe - Japan 300ms

Failover
•Replica sets
•Min 3 nodes
•Master/slave

Failover
•Replica sets
•Min 3 nodes
•Master/slave
•Automatic failover

•Replication lag
Performance
Location Ping RTT Latency
Within USA 40-80ms
Trans-Atlantic 100ms
Trans-Paciﬁc 150ms
Europe - Japan 300ms

Replication Lag
1. Reads: eventual consistency

Replication Lag
1. Reads: eventual consistency
2. Failover: slave behind

Eventual Consistency
Stale data
Inconsistent data

Eventual Consistency
Stale data
Inconsistent data
Changing data

Eventual Consistency
Use Case Needs consistency?
Graphs No
User proﬁle Yes
Statistics Depends
Alert conﬁg Yes

Slave behind
Failover: out of date master

• Safe by default
>>> from pymongo import MongoClient
>>> connection = MongoClient(w=int/str)
Value Meaning
0 Unsafe
1 Primary
2 Primary + x1 secondary
3 Primary + x2 secondaries
MongoDB WriteConcern

$Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})$

${ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags$

Picture is unrelated! Mmm, ice cream.
• Fast network
•More RAM
Performance

http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

More RAM = expensive
Performance
x2 4GB RAM 12 month Prices

EC2 disk/RAM pricing
Performance
$2232/m
$2520/m
$43/m
$295/m

Tips: rand()
•Field names
•Covered indexes

Tips: rand()
•Field names
•Covered indexes
•Collections / databases

Backups
What is the use case?
Fixing user errors?
Point in time restore?
Disaster recovery?

Backups
•Disaster recovery
Age
Offsite
Restore time

Backups
Frequency
Consistency
Veriﬁcation

www.ﬂickr.com/photos/daddo83/3406962115/
Monitoring
•System
Disk i/o
Disk use

www.ﬂickr.com/photos/daddo83/3406962115/
Monitoring
Disk i/o
Disk use
•System
Swap

www.ﬂickr.com/photos/daddo83/3406962115/
Monitoring
Optime
State
•Replication

David Mytton
david@serverdensity.com
@davidmytton
Woop Japan!
blog.serverdensity.com/mongodb

This document discusses MongoDB infrastructure at Server Density. It notes that Server Density uses 27 MongoDB nodes to store 20TB of data per month from their MySQL database. Some key reasons for choosing MongoDB include replication, official drivers, easy deployment, and fast performance out of the box. The document then discusses various MongoDB performance and infrastructure considerations like network throughput, replication lag, failover processes, disk types, backups, and monitoring.

MongoDB's New Aggregation frameworkChris Westin

The document discusses MongoDB's new aggregation framework, which provides a declarative pipeline for performing data aggregation operations on complex documents. The framework allows users to describe a chain of operations without writing JavaScript. It will offer high-performance operators like $match, $project, $unwind, $group, $sort, and computed expressions to reshape and analyze document data without the overhead of JavaScript. The aggregation framework is nearing release and will support sharding by forwarding pipeline operations to shards and combining results.

OSDC 2012 | Scaling with MongoDB by Ross LawleyNETWAYS

MongoDB's architecture features built-in support for horizontal scalability, and high availability through replica sets. Auto-sharding allows users to easily distribute data across many nodes. Replica sets enable automatic failover and recovery of database nodes within or across data centers. This session will provide an introduction to scaling with MongoDB by one of the developers working on the project.

Tweaking performance on high-load projectsDmitriy Dumanskiy

This document discusses optimizing performance for high-load projects. It summarizes the delivery loads and technologies used for several projects including mGage, mobclix and XXXX. It then discusses optimizations made to improve performance, including using Solr for search, Redis for real-time data, Hadoop for reporting, and various Java optimizations in moving to Java 7. Specific optimizations discussed include reducing garbage collection, improving random number generation, and minimizing I/O operations.

Handling 20 billion requests a monthDmitriy Dumanskiy

This document discusses the architecture and technical challenges of handling a large volume of requests for an online advertising platform. It summarizes three key projects handled by the platform that delivered 3 billion, 14 billion, and 20 billion requests per month respectively. It describes the technologies used, including Solr, Redis, MySQL, Hadoop and Amazon Web Services instances. It also outlines optimizations made to improve performance, such as data compression, query optimizations, and Java 7 improvements. The goal was to process over 11,000 requests per second on average while maintaining response times below 100ms.

Understanding and tuning WiredTiger, the new high performance database engine...Ontico

MongoDB 3.0 introduced the concept of different storage engine. The new engine known as WiredTiger introduces document level MVCC locking, compression and a choice between Btree or LSM indexes. In this talk you will learn about the storage engine architecture and specifically WiredTiger, and how to tune and monitor it for best performance. MongoDB 3.0 представил новый концепт движков хранения. Новый движок известен как WiredTiger и предоставляет новый уровень документов MVCC фикс, компрессию и выбор между Btree или индексами LSM. В этом докладе вы поймете, как тюнить и мониторить архитектуры движка базы данных, а точнее WiredTiger для получения максимальной производительности.

Performance Tuning and OptimizationMongoDB

This document provides an overview of performance tuning and optimization in MongoDB. It defines performance tuning as modifying a system to handle increased load, while optimization is modifying a system to work more efficiently or use fewer resources. Measurement tools discussed include log files, the profiler, query optimizer, and explain plans. Effecting change involves measuring current performance, identifying bottlenecks, removing bottlenecks, remeasuring, and repeating. Possible areas for improvement discussed are schema design, access patterns, indexing, hardware configuration, and instance configuration. The document provides examples and best practices around indexing, access patterns, and hardware tuning.

JVM performance options. How it worksDmitriy Dumanskiy

What to know about Amazon Elastic Block Store (EBS)LCloud

This document provides information about Amazon Elastic Block Store (EBS). It begins with an introduction to the author and their experience and certifications. It then discusses EBS as a block storage service from AWS that is network-attached storage for EC2 instances. Key points made include that EBS volumes can be independently created, attached to EC2 instances, resized, and snapshotted for backups. Different EBS volume types like GP2, IO1, ST1, and SC1 are suited for different use cases based on their performance characteristics and cost. Choosing the right volume type depends on factors like throughput needs, IOPS requirements, and whether sequential or random access is more important.

Painless ruby deployment on shelly cloudGiedrius Rimkus

Shelly Cloud is a Platform as a Service (PaaS) that allows for painless Ruby deployments through git pushes. Developers can focus on building applications rather than deployment and infrastructure by using Shelly Cloud to host and run Ruby applications. Shelly Cloud supports Ruby, Rails, Sinatra and other Rack-based applications. It provides features like running periodic tasks, background processing, storing files, caching, sending emails, and more through an easy to use command line interface and cloud management through a Cloudfile.

Webinar Back to Basics 3 - Introduzione ai Replica SetMongoDB

Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Ontico

Write-optimized database algorithms have been available in NoSQL products for many years. With MyRocks, the RocksDB storage engine for MySQL, we are using a write-optimized algorithm for a SQL DBMS. This talk will explain why we created MyRocks and how to compare write-optimized algorithms with the ubiquitous B-Tree in terms of read, write and space efficiency. A large MySQL deployment at Facebook is in the process of migrating from InnoDB to MyRocks. With RocksDB storage engines for MySQL and MongoDB and the Vinyl storage engine for Tarantool we think it is likely that you can consider a write-optimized database engine in the next few years.

Redis: REmote DIctionary ServerEzra Zygmuntowicz

Introduction to RedisMaarten Smeets

Redis is an open source in memory database which is easy to use. In this introductory presentation, several features will be discussed including use cases. The datatypes will be elaborated, publish subscribe features, persistence will be discussed including client implementations in Node and Spring Boot. After this presentation, you will have a basic understanding of what Redis is and you will have enough knowledge to get started with your first implementation!

Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...DataStax Academy

Presenter: Ben Vanberg, Senior Software Engineer at FullContact Here at FullContact we have lots and lots of contact data. In particular we have more than a billion profiles over which we would like to perform ad hoc data analysis. Much of this data resides in Cassandra, and we have many analytics MapReduce jobs that require us to iterate across terabytes of Cassandra data. To solve this problem we've implemented our own splittable input format which allows us to quickly process large SSTables for downstream analytics.

MongoDB and PythonNorberto Leite

MongoDB is one of the most popular databases these days and there are a few reasons for such popularity. One of these reasons is the excellent integration with different programming languages and development frameworks. In the case of Python we take it a few notches up (native use of dictionaries, integration with asynchronous libraries (twisted, gevent), good support for web frameworks like django, flask, bottle ... (mongoengine anyone?). This talk is about the several different projects that we support, the way to effectively use Python and MongoDB together and a few other improvements and announcements.

Back to Basics Webinar 3: Introduction to Replica SetsMongoDB

This document provides an introduction to MongoDB replica sets, which allow for data redundancy and high availability. It discusses how replica sets work, including the replica set life cycle and how applications should handle writes and queries when using a replica set. Specifically, it explains that the MongoDB driver is responsible for server discovery and monitoring, retry logic, and handling topology changes in a replica set to provide a consistent view of the data to applications.

JEEConf. Vanilla javaDmitriy Dumanskiy

This document discusses various approaches to building high performance Java applications for handling IoT workloads. It begins by noting that vanilla Java can handle 10,000 requests per second per CPU core. It then discusses blocking and non-blocking IO approaches, highlighting several non-blocking frameworks like Netty. The document also summarizes the architecture and performance of the Blynk IoT platform, which uses a non-blocking architecture with batches, no synchronization, and in-memory structures to achieve high throughput. Finally, it outlines different versions of the Blynk platform with increasing scale and resilience.

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

MongoDB is easy to download and run locally but requires some thought and further understanding when deploying to production. At scale, schema design, indexes and query patterns really matter. So does data structure on disk, sharding, replication and data centre awareness. This talk will examine these factors in the context of analytics, and more generally, to help you optimise MongoDB for any scale. Presented at MongoDB Days London 2013 by David Mytton.

Mastering the MongoDB ShellMongoDB

This document discusses the MongoDB shell, which is a JavaScript interpreter with built-in support for MongoDB. It can be used for interactive development, testing, administration, and learning MongoDB. The shell allows running queries, saving documents, and has features like command completion, history, and helpers for common tasks. While powerful, it has some limitations like performance and lack of modern JavaScript features. Overall, the document recommends using the shell to better understand and work with MongoDB.

ETL With Cassandra Streaming Bulk Loadingalex_araujo

Cassandra ETL uses Chef recipes to configure Cassandra clusters on EC2 for bulk loading data. A custom Java ETL JAR processes input files to generate SSTables, which are streamed into Cassandra using sstableloader for fast loading. The Grinder is used to stress test and measure the import performance and throughput across the Cassandra cluster. Results showed streaming bulk loads were 2.5x faster than Thrift and up to 300x faster than MySQL. The only downside was the custom SSTable generation was slower than Cassandra's native writes.

Replication and replica setsRandall Hunt

This document discusses MongoDB replication and replica sets. It begins with an overview of why replication is useful, including protecting against node failures, network latency, and having different uses for data. It then covers the lifecycle of a replica set from creation to recovery. It discusses the roles nodes can have in a replica set and how configurations are set. It explains how to develop applications using replica sets, including considerations for strong vs. delayed consistency, write concerns, tagging data, and read preferences. Finally, it discusses some operational considerations for replica sets like maintenance, upgrades, and deployment architectures for single vs. multiple data centers.

Azure sql insert perfMornè Blake

Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...Ontico

NoSQL key-value — популярное решение, но какие преимущества и какой ценой оно даёт? Скорость? Возможно, но ценой урезанного, по сравнению с реляционными базами данных, функционала. Но данные и приложение всё еще разделены сетевым стеком, а иногда и десятками километров оптоволокна. В крупном проекте, работающем на десятках или сотнях серверов нельзя обеспечить высокую скорость доступа к данным с каждой машины. Если хранилище достаточно быстро, то время обработки запроса окажется значительно меньше затрат на работу с сетью, а производительность приложения будут определять сетевые задержки и частота запросов к БД. В проекте Облако@Mail.Ru мы ушли от использования чистых key-value хранилищ в пользу микросервисов на Tarantool, что позволило свести общение с хранилищем данных к минимуму. Да, Tarantool — это еще одна NoSQL база данных, но еще это полноценный сервер приложений. Приложений, расположенных рядом с данными! Я расскажу, как мы пришли к использованию микросервисов на основе Tarantool. Приведу несколько сценариев использования, которые работают в Облаке и могут быть легко адаптированы для другого web-проекта. Вы узнаете о компонентах, которые разработаны и опубликованы нами уже сейчас, и о дальнейших планах развития.

Python and MongoDB Norberto Leite

Back to Basics Webinar 6: Production DeploymentMongoDB

Background Tasks in Node - Evan Tahler, TaskRabbitRedis Labs

The part of the talk discusses various strategies for handling background tasks in Node.js, including: 1. Foreground tasks, which process tasks inline but can slow down response times. 2. Parallel tasks, which ignore callbacks to try and speed up response times but lose error handling. 3. Local messages using clustering, which uses inter-process communication to pass tasks to worker processes and provides error handling but is limited to one server. 4. Remote messages using Redis pub/sub, which allows distributing tasks across multiple servers but limits server types to one. 5. Remote queue using Node-Resque, which treats tasks as jobs in a Redis queue for scheduling across worker processes, providing

Whats new in mongoDB 2.4 at Copenhagen user group 2013-06-19Henrik Ingo

Navigating the customer lifecycleServer Density

Puppet Camp Ghent 2013Server Density

This document discusses adding Forge modules to Puppet Enterprise. It describes moving the HTTP load balancer from Pound to nginx, keeping existing manifests pulled from GitHub, and using the Puppet Forge to install the puppetlabs/nginx module or integrating it via Git submodules. It also covers parameterizing classes on the Puppet Enterprise console and merging site.pp files, as well as updating nginx configurations on the fly and considering alternative nginx modules on the Forge.

More Related Content

What's hot (20)

What to know about Amazon Elastic Block Store (EBS)LCloud

Painless ruby deployment on shelly cloudGiedrius Rimkus

Webinar Back to Basics 3 - Introduzione ai Replica SetMongoDB

Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Ontico

Redis: REmote DIctionary ServerEzra Zygmuntowicz

Introduction to RedisMaarten Smeets

Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...DataStax Academy

MongoDB and PythonNorberto Leite

Back to Basics Webinar 3: Introduction to Replica SetsMongoDB

JEEConf. Vanilla javaDmitriy Dumanskiy

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

Mastering the MongoDB ShellMongoDB

ETL With Cassandra Streaming Bulk Loadingalex_araujo

Replication and replica setsRandall Hunt

Azure sql insert perfMornè Blake

Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...Ontico

Python and MongoDB Norberto Leite

Back to Basics Webinar 6: Production DeploymentMongoDB

Background Tasks in Node - Evan Tahler, TaskRabbitRedis Labs

Whats new in mongoDB 2.4 at Copenhagen user group 2013-06-19Henrik Ingo

What to know about Amazon Elastic Block Store (EBS)LCloud

Painless ruby deployment on shelly cloudGiedrius Rimkus

Webinar Back to Basics 3 - Introduzione ai Replica SetMongoDB

Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Ontico

Redis: REmote DIctionary ServerEzra Zygmuntowicz

Introduction to RedisMaarten Smeets

Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...DataStax Academy

MongoDB and PythonNorberto Leite

Back to Basics Webinar 3: Introduction to Replica SetsMongoDB

JEEConf. Vanilla javaDmitriy Dumanskiy

MongoDB: Optimising for Performance, Scale & AnalyticsServer Density

Mastering the MongoDB ShellMongoDB

ETL With Cassandra Streaming Bulk Loadingalex_araujo

Replication and replica setsRandall Hunt

Azure sql insert perfMornè Blake

Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...Ontico

Python and MongoDB Norberto Leite

Back to Basics Webinar 6: Production DeploymentMongoDB

Background Tasks in Node - Evan Tahler, TaskRabbitRedis Labs

Whats new in mongoDB 2.4 at Copenhagen user group 2013-06-19Henrik Ingo

Viewers also liked (19)

Navigating the customer lifecycleServer Density

Puppet Camp Ghent 2013Server Density

StartOps: Growing an ops team from 1 founderServer Density

Bootstrapped startups don't have the luxury of a full team of ops engineers available to respond to issues 24/7, so how can you survive on your own? This talk will tell the story of how to run your infrastructure as a single founder through to growing that into a team of on call engineers. It will include some interesting war stories as well as tips and suggestions for how to run ops at a startup. Presented at DevOpsDays London 2013 by David Mytton.

High performance Infrastructure Oct 2013Server Density

The document discusses high performance infrastructure for Server Density which includes 150 servers that have been running since June 2009 and migrated from MySQL to MongoDB. It stores 25TB of data per month. Key aspects of performance discussed are using fast networks like 10 Gigabit Ethernet on AWS, ensuring high memory, using SSDs over spinning disks for performance, and factors like replication lag based on location. The document also compares options like using cloud, dedicated servers, or colocation and discusses monitoring, backups, dealing with outages, and other operational aspects.

Experiences from DevOps production: Deployment, performance, failure.Server Density

Remote startup - building a company from everywhere in the worldServer Density

This document discusses how Server Density grew from 2 employees in 2009 to 12 employees in 2013 while being fully remote. It outlines the company's timeline and some advantages of being fully remote such as access to worldwide talent, lower costs, and no distractions of an office. However, it also notes disadvantages like collaboration being more difficult without in-person interactions. While the company added an office in 2012, the document emphasizes that working remotely is a mindset that is difficult to adopt after initially co-locating. Effective communication and availability are important for fully distributed teams.

Handling incidentsServer Density

The document discusses how to handle incidents, downtime, and outages. It notes that the cost of downtime for companies in Q1 2015 was $2.9 billion, $870 million, and $4.1 billion. It recommends preparing for incidents by having on-call staff and documentation, responding quickly by following an incident response checklist and notifying stakeholders, and performing a postmortem within days to analyze what failed and how to prevent future issues.

Puppet Camp 2012Server Density

Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density. Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed. Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances. www.serverdensity.com/puppetcamp/

Briefing: ContainersServer Density

Containers seem to have suddenly become the hot new thing everyone is talking about, but what are they? Why are they important? How should you use them and what does it mean for cloud infrastructure? This talk will examine the history, technical details and strategy around containerisation from the perspective of developers and operations, consider internal container OSs like Rocket and Ubuntu Core as well as management layers like Docker and Apache Mesos and take a look at why cloud providers are launching their own services around them. Presented by David Mytton at Datacloud Monaco 2015-06-04

Scaling humans - Ops teams and incident managementServer Density

The document discusses the costs of downtime for companies and best practices for incident management teams to prepare for, respond to, and review incidents to minimize downtime costs. It notes that downtime costs companies billions per quarter and recommends teams prepare documentation and contact information, have on-call rotation schedules, log all responses to incidents, provide frequent status updates, gather teams to escalate incidents as needed, and conduct post-mortem reviews within days of an incident.

DevOps Incident Handling - Making friends not enemies.Server Density

Why puppet? Why now?Server Density

Why Puppet? Why now? Can you get by without using any config management? You probably think don't have time, or that your project is too small. What can using Puppet really add? How can you justify investing time up front? Maybe you can just do it later? Getting started with config management can often seem like a big project, especially if you only manage a few systems or have a small team. This talk will examine why you should use Puppet from the beginning. It will examine what you can do with Puppet that couldn't do otherwise, how much time it will save and why it's especially important if you think your project has even the smallest chance of scaling in the future. Presented by David Mytton at Puppet Camp London 2015-04-13

Puppet at the centre of everythingServer Density

The document discusses Server Density's architecture which includes 100 Ubuntu servers with 50% being virtual, using Nginx, Python, and MongoDB. It handles 25TB of data per month. Puppet is used for configuration, failover, code deploys, and system updates. The document also considers colocating servers versus using a dedicated provider and factors like hardware specs, costs, skills required, and fun.

Infrastructure choices - cloud vs colo vs bare metalServer Density

This document discusses the differences between cloud, colocation, and bare metal infrastructure options. It covers key considerations for performance including CPU, memory, disk, and network latency and bandwidth. Colocation provides hardware at a specific location while maintaining internal skills, but has costs for total spend, hardware specifications, and power usage. Cloud infrastructure offers elastic workloads and support for demand spikes and unknown requirements, but bare metal is preferable for managed hardware replacement and networking needs. Overall, the best option depends on an organization's specific workload characteristics and skills.

Flight training for DevOps & HumanOps - IncontroDevOps 2016Server Density

Flight training for DevOpsServer Density

This document discusses improving incident response procedures through practices such as checklists, documented procedures, realistic incident simulations and postmortems. It recommends extended use of checklists to guide responses while still allowing for experience and independent thought. Regular incident response simulations that test both general processes and specific failures can help refine procedures and build confidence. Postmortems should objectively review incidents, suggest improvements and run through scenarios again over time to prevent complacency.

Content marketing @ Server DensityServer Density

How to monitor NGINXServer Density

How to Monitor MySQLServer Density

Navigating the customer lifecycleServer Density

Puppet Camp Ghent 2013Server Density

StartOps: Growing an ops team from 1 founderServer Density

High performance Infrastructure Oct 2013Server Density

Experiences from DevOps production: Deployment, performance, failure.Server Density

Remote startup - building a company from everywhere in the worldServer Density

Handling incidentsServer Density

Puppet Camp 2012Server Density

Briefing: ContainersServer Density

Scaling humans - Ops teams and incident managementServer Density

DevOps Incident Handling - Making friends not enemies.Server Density

Why puppet? Why now?Server Density

Puppet at the centre of everythingServer Density

Infrastructure choices - cloud vs colo vs bare metalServer Density

Flight training for DevOps & HumanOps - IncontroDevOps 2016Server Density

Flight training for DevOpsServer Density

Content marketing @ Server DensityServer Density

How to monitor NGINXServer Density

How to Monitor MySQLServer Density

Similar to NoSQL Infrastructure - Late 2013 (20)

Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...MongoDB

MongoDB for Time Series Data Part 3: ShardingMongoDB

The document discusses sharding time series sensor data in MongoDB. It recommends modeling the application's read, write and storage patterns to determine the optimal sharding strategy. A good shard key has sufficient cardinality, distributes writes evenly and enables targeted reads. For time series data, a compound shard key of an arbitrary value and incrementing timestamp is suggested to balance hot spots and targeted queries. The document also covers configuring a sharded cluster and replica sets with tags to control data distribution.

Mongo db roma replication and shardingGuglielmo Incisa Di Camerana

Replication and sharding allow MongoDB databases to scale horizontally across commodity servers for high availability and increased performance. Replication duplicates data across multiple nodes so the application can continue running if a node fails. Sharding partitions data across nodes to distribute load and storage requirements. MongoDB supports combining replication within shards for redundancy and sharding to partition data across replicated shards, providing redundancy and scalability.

Nodejs - A quick tour (v6)Felix Geisendörfer

This document provides an overview and introduction to Node.js. It discusses that Node.js is a platform for building scalable network applications using JavaScript and uses non-blocking I/O and event-driven architecture. It was created by Ryan Dahl in 2009 and uses Google's V8 JavaScript engine. Node.js allows building web servers, networking tools and real-time applications easily and efficiently by handling concurrent connections without threads. Some popular frameworks and modules built on Node.js are also mentioned such as Express.js, Socket.IO and over 1600 modules in the npm registry.

Mongodb workshopHarun Yardımcı

This document discusses MongoDB replication and sharding. It begins with an introduction to MongoDB and its data model. It then covers replication with master-slave and replica sets. It also discusses sharding components including shard servers, config servers, and mongos routers. It provides examples of setting up replica sets, adding shards, and combining replica sets with sharding. The document encourages trying these techniques yourself and sharing experiences.

MongoDBSteven Francia

Functional Hostnames and Why they are BadPuppet

Nodejs - A quick tour (v5)Felix Geisendörfer

Node.js is a JavaScript runtime built on Chrome's V8 engine. It allows JavaScript to be run on the server-side. Node.js avoids blocking I/O operations by using non-blocking techniques and event loops. It provides APIs for common tasks like HTTP servers, filesystem access, and more. While still in development, Node.js has found success in building real-time applications and APIs due to its asynchronous and non-blocking architecture.

Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Muhammad Ali

2013 london advanced-replicationMarc Schwering

This document discusses advanced replication in MongoDB replica sets. It begins by explaining the roles and configuration of replica set members, including primaries, secondaries, and arbiters. It then covers the implementation details of replication using an oplog and heartbeats between nodes. The rest of the document discusses best practices for maintenance, data center awareness using tagging, write concerns, and read preferences to control data distribution and failover.

The powerful toolset of the go-mysql libraryDaniël van Eeden

Maintenance for MongoDB Replica SetsIgor Donchovski

Redundancy and high availability are the basis for all production deployments. With MongoDB high availability is achieved with replica sets which provides automatic fail-over in case the Primary goes down. In this session we will review multiple maintenance scenarios that will include the proper steps for keeping the high availability while we perform the maintenance steps without causing downtime. This session will cover Database upgrades, OS server patching, Hardware upgrades, Network maintenance and more. How MongoDB HA works Replica sets components/deployment typologies Database upgrades System patching/upgrade Network maintenance Add/Remove members to the replica set Reconfiguring replica set members Building indexes Backups and restores

Cutting through the fog of cloudKyle Rames

Mongodb introchristkv

Christian Kvalheim gave an introduction to NoSQL and MongoDB. Some key points: 1) MongoDB is a scalable, high-performance, open source NoSQL database that uses a document-oriented model. 2) It supports indexing, replication, auto-sharding for horizontal scaling, and querying. 3) Documents are stored in JSON-like records which can contain various data types including nested objects and arrays.

Tweaking perfomance on high-load projects_Думанский ДмитрийGeeksLab Odessa

This document discusses optimizing the performance of several high-load projects delivering billions of requests per month. It summarizes the evolution and delivery loads of different projects over time. It then analyzes the technical stacks and architectures used, identifying problems and solutions implemented around areas like querying, data storage, processing, and networking. Key lessons learned are around sharding and resharding data, optimizing I/O, using streaming processing like Storm over batch processing like Hadoop, and working within AWS limits and capabilities.

Squeak DBXESUG

SqueakDBX is an open-source solution for providing relational database access from Smalltalk. It aims to allow direct SQL queries to any database using a common API, avoiding the limitations of existing Smalltalk database drivers. SqueakDBX uses an asynchronous C connector called OpenDBX to provide fast and flexible access to multiple databases without locking the Smalltalk virtual machine. The project seeks to fully integrate SqueakDBX with the GLORP object-relational mapping library to improve database usage in Smalltalk.

Latinowarekchodorow

The document discusses MongoDB, a non-relational database that uses documents with a flexible schema rather than tables, and is well-suited for applications that need to store data in complex, nested structures. It provides an overview of key MongoDB concepts like collections, queries, indexing, updating documents, and aggregation capabilities. Examples are given of how MongoDB can be used for applications involving user profiles, blogs, and logging.

Nodejs - Should Ruby Developers Care?Felix Geisendörfer

Node.js is a platform for building scalable network applications. It uses Google's V8 JavaScript engine and a non-blocking I/O model. Some key points: - Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, especially for real-time applications. - It has a large ecosystem of open source modules. Popular frameworks include Express and Fab. - While Node.js is very fast for I/O operations, memory usage can grow quickly and scaling to multiple cores requires multiple processes. - The author argues Node.js is suitable for single-page apps, real-time applications, and crawlers, but

[AWS Builders] Effective AWS GlueAmazon Web Services Korea

NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!Daniel Cousineau

Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...MongoDB

MongoDB for Time Series Data Part 3: ShardingMongoDB

Mongo db roma replication and shardingGuglielmo Incisa Di Camerana

Nodejs - A quick tour (v6)Felix Geisendörfer

Mongodb workshopHarun Yardımcı

MongoDBSteven Francia

Functional Hostnames and Why they are BadPuppet

Nodejs - A quick tour (v5)Felix Geisendörfer

Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Muhammad Ali

2013 london advanced-replicationMarc Schwering

The powerful toolset of the go-mysql libraryDaniël van Eeden

Maintenance for MongoDB Replica SetsIgor Donchovski

Cutting through the fog of cloudKyle Rames

Mongodb introchristkv

Tweaking perfomance on high-load projects_Думанский ДмитрийGeeksLab Odessa

Squeak DBXESUG

Latinowarekchodorow

Nodejs - Should Ruby Developers Care?Felix Geisendörfer

[AWS Builders] Effective AWS GlueAmazon Web Services Korea

NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!Daniel Cousineau

Recently uploaded (20)

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Want to learn practical tips for designing systems that can scale efficiently without compromising speed? Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development. As you explore key principles of designing low-latency systems with Rust, you will learn how to: - Create and compile a real-world app with Rust - Connect the application to ScyllaDB (NoSQL data store) - Negotiate tradeoffs related to data modeling and querying - Manage and monitor the database for consistently low latencies

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/ HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client. Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience. In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including - Accessing the console - Locating and interpreting log files - Accessing the data folder within the browser’s cache (using OPFS) - Understand the difference between single- and multi-user scenarios - Utilizing Client Clocking

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Building 10x Organizations with Modern Productivity Metrics 10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’ Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them. But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

tecnologias de las primeras civilizaciones.pdffjgm517

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock

Big Data Analytics Quick Research Guide by Arthur MorganArthur Morgan

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

tecnologias de las primeras civilizaciones.pdffjgm517

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Enhancing ICU Intelligence: How Our Functional Testing Enabled a Healthcare I...Impelsys Inc.

NoSQL Infrastructure - Late 2013

1. NoSQL Infrastructure

2. ~3333 write ops/s 0.07 - 0.05 ms response

3. David Mytton Woop Japan!

5. MongoDB at Server Density

6. •27 nodes MongoDB at Server Density

7. • June 2009 - 4yrs •27 nodes MongoDB at Server Density

8. •MySQL -> MongoDB •27 nodes MongoDB at Server Density • June 2009 - 4yrs

9. •MySQL -> MongoDB •25TB data per month •27 nodes MongoDB at Server Density • June 2009 - 4yrs

10. Why?

11. Why? • Replication

12. Why? • Replication • Ofﬁcial drivers

13. Why? • Replication • Ofﬁcial drivers • Easy deployment

14. Why? • Replication • Ofﬁcial drivers • Easy deployment • Fast out of the box

15. It’s a little different.

16. Picture is unrelated! Mmm, ice cream. • Fast network Performance

17. • Fast network Performance EC2 10 Gigabit Ethernet - Cluster Compute - High Memory Cluster - Cluster GPU - High I/O - High Storage

18. • Fast network Performance Workload: Read/Write? Result set size What is being stored?

19. • Fast network Performance Use Network Throughput Normal 0-100Mb/s Replication (Initial Sync) Burst +100Mb/s Replication (Oplog) 0-100Mb/s Backup Initial Sync + Oplog

20. • Fast network Performance Inter-DC LAN

21. • Fast network Performance Inter-DC LAN Cross USA Washington, DC - San Jose, CA

22. • Fast network Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Paciﬁc 150ms Europe - Japan 300ms

23. Failover •Replica sets

24. Failover •Replica sets •Master/slave

25. Failover •Replica sets •Min 3 nodes •Master/slave

26. Failover •Replica sets •Min 3 nodes •Master/slave •Automatic failover

27. •Replication lag Performance Location Ping RTT Latency Within USA 40-80ms Trans-Atlantic 100ms Trans-Paciﬁc 150ms Europe - Japan 300ms

28. Replication Lag 1. Reads: eventual consistency

29. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind

30. Eventual Consistency Stale data

31. Eventual Consistency Stale data Inconsistent data

32. Eventual Consistency Stale data Inconsistent data Changing data

33. Eventual Consistency Use Case Needs consistency? Graphs No User proﬁle Yes Statistics Depends Alert conﬁg Yes

34. Replication Lag 1. Reads: eventual consistency 2. Failover: slave behind

35. Slave behind Failover: out of date master

36. • Safe by default >>> from pymongo import MongoClient >>> connection = MongoClient(w=int/str) Value Meaning 0 Unsafe 1 Primary 2 Primary + x1 secondary 3 Primary + x2 secondaries MongoDB WriteConcern

38. Tags { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"})

39. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "veryImportant"}) (A or B) + (C or D) + E Tags

40. { _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } } } > db.foo.insert({x:1}) > db.runCommand({getLastError : 1, w : "sortOfImportant"}) (A + C) or (D + E) ... Tags

41. Picture is unrelated! Mmm, ice cream. • Fast network •More RAM Performance

42. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs

43. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

44. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html

45. More RAM = expensive Performance x2 4GB RAM 12 month Prices

46. RAM SSDs Spinning disk Cost Speed

47. Softlayer disk pricing Performance

48. EC2 disk/RAM pricing Performance $2232/m $2520/m $43/m $295/m

49. SSD vs Spinning Performance

50. SSD vs Spinning Performance

51. SSD vs Spinning Performance

52. Tips: rand() •Field names