Key-Value-Stores -- The Key to Scaling?

Oct 20, 201010 likes6,433 views

The document discusses key-value stores as options for scaling the backend of a Facebook game. It describes Redis, Cassandra, and Membase and evaluates them as potential solutions. Redis is selected for its simplicity and ability to handle the expected write-heavy workload using just one or two servers initially. The game has since launched and is performing well with the Redis implementation.

Key-Value-Stores:
The Key to Scaling?
Tim Lossen

Who?
• @tlossen
• backend developer
- Ruby, Rails, Sinatra ...
• passionate about technology

Key-Value-Stores -- The Key to Scaling?

Key-Value-Stores -- The Key to Scaling?

Problem

Challenge
• backend for facebook game

Challenge
• backend for facebook game
• expected load:
- 1 mio. daily active users
- 20 mio. total users
- 100 KB data per user

Challenge
• expected peak traﬃc:
- 10.000 concurrent users
- 200.000 requests / minute

Challenge
• expected peak traﬃc:
- 10.000 concurrent users
- 200.000 requests / minute
• write-heavy workload

Wanted
• scalable database
• with high throughput
- especially for writes

Options
• relational database
- with sharding

Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

Shortlist
• Cassandra
• Redis
• Membase

Cassandra

Facts
• written in Java
- 55.000 lines of code
• Thrift API
- clients for Java, Ruby, Python ...

History
• originally developed by Facebook
- in production for “Inbox Search”
• later open-sourced
- top-level Apache project

Features
• high availability
- no single point of failure
• incremental scalability
• eventual consistency

Architecture
• Dynamo-like hash ring
- partitioning + replication
- all nodes are equal

Hash Ring

Architecture
• Dynamo-like hash ring
- partitioning + replication
- all nodes are equal
• Bigtable data model
- column families
- supercolumns

Key-Value-Stores -- The Key to Scaling?

“Cassandra aims to run
on an in"astructure
of hundreds of nodes.”

Redis

Facts
• written in C
- 13.000 lines of code
• socket API
- redis-cli
- client libs for all major languages

Features
• high read & write throughput
- 50.000 to 100.000 ops / second

Features
• high read & write throughput
- 50.000 to 100.000 ops / second
• interesting data structures
- lists, hashes, (sorted) sets
- atomic operations

Features
• high read & write throughput
- 50.000 to 100.000 ops / second
• interesting data structures
- lists, hashes, (sorted) sets
- atomic operations
• strong consistency

Architecture
• in-memory database
- append-only log on disk
- virtual memory

Architecture
• in-memory database
- append-only log on disk
- virtual memory
• single instance
- master-slave replication
- clustering is on roadmap

“Memory is the new disk,
disk is the new tape.”
— Jim Gray

Membase

Facts
• written in C and Erlang
• API-compatible to Memcached
- same protocol
• client libs for all major languages

History
• developed by NorthScale & Zynga
- used in production (Farmville)
• released in June 2010
- Apache 2.0 License

Features
• “Memcached with persistence”
- extremely fast
- throughput scales linearly

Features
• “Memcached with persistence”
- extremely fast
- throughput scales linearly
• automatic data placement
- memory, ssd, disk

Features
• “Memcached with persistence”
- extremely fast
- throughput scales linearly
• automatic data placement
- memory, ssd, disk
• conﬁgurable replica count

Architecture
• cluster
- all nodes are alike
- one elected as “coordinator”

Architecture
• cluster
- all nodes are alike
- one elected as “coordinator”
• each node is master for part of key
space
- handles all reads & writes

Mapping Scheme

“simple, fast, elastic”

Solution

Which one
would you pick?

Decision
• Cassandra ?

Decision
• Cassandra ?
- too big, too complicated

Decision
• Cassandra ?
- too big, too complicated
• Membase ?

Decision
• Cassandra ?
- too big, too complicated
• Membase ?
- not yet available (then)

Decision
• Cassandra ?
- too big, too complicated
• Membase ?
- not yet available (then)
• Redis !

Motivation
• keep operations simple
• use as few machines as possible
- ideally, only one

Design
• two machines (+ load balancer)
- Redis master handles all reads /
writes
- Redis slave as hot standby

Design
• two machines (+ load balancer)
- Redis master handles all reads /
writes
- Redis slave as hot standby
- both machines used as app servers

Design
• two machines (+ load balancer)
- Redis master handles all reads /
writes
- Redis slave as hot standby
- both machines used as app servers
• dedicated hardware

Data model
• one Redis hash per user
- key: facebook id
• store data as serialized JSON
- booleans, strings, numbers,
timestamps ...

Advantages
• turns Redis into “document db”
- eﬃcient to swap user data in / out
- atomic ops on parts
• easy to dump / restore user data

Capacity
• 4 GB memory for 20 mio. integer keys
- keys always stay in memory!

Capacity
• 4 GB memory for 20 mio. integer keys
- keys always stay in memory!
• 2 GB memory for 10.000 user hashes
- others can be swapped out

Capacity
• 4 GB memory for 20 mio. integer keys
- keys always stay in memory!
• 2 GB memory for 10.000 user hashes
- others can be swapped out
• 3.6 mio. ops / minute
- suﬃcient for 200.000 requests

Status
• game was launched in august
- currently still in beta

Status
• game was launched in august
- currently still in beta
• expect to reach 1 mio. daily active users
in Q1/2011

Status
• game was launched in august
- currently still in beta
• expect to reach 1 mio. daily active users
in Q1/2011
• will try to stick to 2 or 3 machines
- possibly bigger / faster ones

Conclusions
• use the right tool for the job

Conclusions
• use the right tool for the job
• keep it simple
- avoid sharding, if possible

Conclusions
• use the right tool for the job
• keep it simple
- avoid sharding, if possible
• don’t scale out too early
- but have a viable “plan b”

Conclusions
• use the right tool for the job
• keep it simple
- avoid sharding, if possible
• don’t scale out too early
- but have a viable “plan b”
• use dedicated hardware

Q&A

Links
• cassandra.apache.org
• redis.io
• membase.org

• tim.lossen.de

Recommended

Cassandra vs. Redis

Cassandra vs. Redis

Cassandra vs. RedisTim Lossen

This document compares Cassandra and Redis for use as a backend for a Facebook game with 1 million daily users and 10 million total users. Redis was chosen over Cassandra due to its simpler architecture, higher write throughput, and ability to meet the capacity and performance requirements using a single node. The Redis master handled all reads and writes, with a slave for failover. User data was stored in Redis hashes to turn it into a "document DB" and allow for atomic operations on parts of the data.

Boosting Machine Learning with Redis Modules and Spark

Boosting Machine Learning with Redis Modules and Spark

Boosting Machine Learning with Redis Modules and SparkDvir Volk

Redis modules allow for new capabilities like machine learning models to be added to Redis. The Redis-ML module stores machine learning models like random forests and supports operations like model training, evaluation, and prediction directly from Redis for low latency. Spark can be used to train models which are then saved as Redis modules, allowing models to be easily deployed and accessed from services and clients.

Introduction to Redis

Introduction to Redis

Introduction to RedisArnab Mitra

Redis is an open source, in-memory data structure store that can be used as a database, cache, or message broker. It supports data structures like strings, hashes, lists, sets, sorted sets with ranges and pagination. Redis provides high performance due to its in-memory storage and support for different persistence options like snapshots and append-only files. It uses client/server architecture and supports master-slave replication, partitioning, and failover. Redis is useful for caching, queues, and other transient or non-critical data.

HBaseCon 2015- HBase @ Flipboard

HBaseCon 2015- HBase @ Flipboard

HBaseCon 2015- HBase @ FlipboardMatthew Blair

NoSQL: Cassadra vs. HBase

NoSQL: Cassadra vs. HBase

NoSQL: Cassadra vs. HBaseAntonio Severien

This document provides an overview of distributed databases and the Yahoo! Cloud Serving Benchmark (YCSB). It discusses NoSQL databases Cassandra and HBase and how YCSB can be used to benchmark their performance. Experiments were conducted on Amazon EC2 using YCSB to load data and run workloads on Cassandra and HBase clusters. The results showed Cassandra had lower latency and higher throughput than HBase. YCSB provides a way to compare the performance of different databases.

What every developer should know about database scalability, PyCon 2010

What every developer should know about database scalability, PyCon 2010

What every developer should know about database scalability, PyCon 2010jbellis

The document discusses the challenges of scaling databases and provides strategies for improving scalability. It covers: 1) The two main types of database operations that need to be scaled are reads and writes. Common "band-aid" approaches that don't actually scale the database are discussed. 2) Scaling reads can be improved through caching, partitioning caches, and replication. Scaling writes is more difficult and requires approaches like write-back caching, asynchronous replication, and data partitioning. 3) Partitioning data through sharding or vertical partitioning distributes the load across multiple database nodes. Consistent hashing is presented as a method to determine which server owns each data key. The challenges of non-transparent partitioning approaches are also

HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera

HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera

HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, ClouderaCloudera, Inc.

Apache HDFS, the file system on which HBase is most commonly deployed, was originally designed for high-latency high-throughput batch analytic systems like MapReduce. Over the past two to three years, the rising popularity of HBase has driven many enhancements in HDFS to improve its suitability for real-time systems, including durability support for write-ahead logs, high availability, and improved low-latency performance. This talk will give a brief history of some of the enhancements from Hadoop 0.20.2 through 0.23.0, discuss some of the most exciting work currently under way, and explore some of the future enhancements we expect to develop in the coming years. We will include both high-level overviews of the new features as well as practical tips and benchmark results from real deployments.

High Performance Weibo QCon Beijing 2011

High Performance Weibo QCon Beijing 2011

High Performance Weibo QCon Beijing 2011Tim Y

Dissecting Scalable Database Architectures

Dissecting Scalable Database Architectures

Dissecting Scalable Database Architectureshypertable

M|18 How to use MyRocks with MariaDB Server

M|18 How to use MyRocks with MariaDB Server

M|18 How to use MyRocks with MariaDB ServerMariaDB plc

MyRocks in MariaDB summarizes MyRocks, a storage engine for MariaDB that is based on RocksDB. It discusses how MyRocks addresses some of the limitations of InnoDB such as high write and space amplification. It provides details on installing and using MyRocks, including data loading techniques, tuning considerations, and replication support. Parallel replication is supported, but the highest isolation level is repeatable-read and row-based replication must be used.

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDBAthiq Ahamed

This document provides a summary of a presentation that benchmarked the performance of three popular NoSQL databases: Apache Cassandra, Apache HBase, and MongoDB. It describes the architectures and data models of each database. Benchmark tests were run using the Yahoo Cloud Serving Benchmark and found that Apache Cassandra consistently outperformed the other databases across different workloads in terms of load time, read and write performance, and latency. The presentation emphasizes the importance of benchmarks for evaluating NoSQL database performance and choosing the right database based on application requirements.

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...slashn

The document summarizes the evolution of Flipkart's website architecture from 2007 to 2012. Key issues addressed included slow website performance due to slow database queries, isolating reads from writes, isolating production traffic from analytics jobs, implementing caching which introduced complexity, isolating the impact of slow external services, handling spikes in traffic, and separating systems to isolate internal from external requests. The evolution involved learning lessons around scaling databases, isolating systems, managing caching complications, and ensuring systems are not overloaded.

微博cache设计谈Tim Y

The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs like followers and following lists. The caches aim to optimize for fast retrieval of recent home timelines and individual user feeds. Mutexes are used to synchronize cache updates between servers.

Digital Library Collection Management using HBase

Digital Library Collection Management using HBase

Digital Library Collection Management using HBaseHBaseCon

Cassandra Introduction & Features

Cassandra Introduction & Features

Cassandra Introduction & FeaturesPhil Peace

Cassandra is an open source, distributed, decentralized, and fault-tolerant NoSQL database that is highly scalable and provides tunable consistency. It was created at Facebook based on Amazon's Dynamo and Google's Bigtable. Cassandra's key features include elastic scalability through horizontal partitioning, high availability with no single point of failure, tunable consistency levels, and a column-oriented data model with a CQL interface. Major companies like eBay, Netflix, and Apple use Cassandra for applications requiring large volumes of writes, geographical distribution, and evolving data models.

Intro to big data choco devday - 23-01-2014

Intro to big data choco devday - 23-01-2014

Intro to big data choco devday - 23-01-2014Hassan Islamov

This document provides an introduction to big data and Hadoop. It discusses the growth of data from 2006 to 2020. It then introduces key concepts of Hadoop including HDFS, MapReduce, and the Hadoop ecosystem. It describes how HDFS stores and processes large datasets in a distributed manner through block storage on datanodes and metadata management by the namenode. MapReduce provides a programming model for distributed processing of large datasets across clusters. The document also discusses challenges of hardware failures and solutions in Hadoop like HDFS high availability and federation.

HBase: Extreme Makeover

HBase: Extreme Makeover

HBase: Extreme MakeoverHBaseCon

Speaker: Vladimir Rodionov (bigbase.org) This talks introduces a totally new implementation of a multilayer caching in HBase called BigBase. BigBase has a big advantage over HBase 0.94/0.96 because of an ability to utilize all available server RAM in the most efficient way, and because of a novel implementation of a L3 level cache on fast SSDs. The talk will show that different type of caches in BigBase work best for different type of workloads, and that a combination of these caches (L1/L2/L3) increases the overall performance of HBase by a very wide margin.

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?Accumulo Summit

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed Storage

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed Storage

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed StorageCloudera, Inc.

Pinterest uses Apache HBase to store data for users' personalized "following feeds" at scale. This involves storing billions of pins and updates per day. Some key challenges addressed are handling high throughput writes from fanouts, providing low latency reads, and resolving potential data inconsistencies from race conditions. Optimizations to HBase include increased memstore size, block cache tuning, and prefix compression. Maintaining high availability involves writing to dual clusters, tight Zookeeper timeouts, and automated repairs.

MyDUMPER : Faster logical backups and restores

MyDUMPER : Faster logical backups and restores

MyDUMPER : Faster logical backups and restores Mydbops

This document discusses using mydumper and myloader tools to perform faster logical backups and restores of MySQL databases. Mydumper is an open source backup tool that can backup databases in parallel threads, split output into multiple files, compress files, and support selective backups of databases and tables. It has advantages over the built-in mysqldump such as being multithreaded, supporting compression, and ability to chunk output. The document provides details on mydumper features, options, and examples of using it for full and partial backups. It also introduces myloader for restoring mydumper backups and provides an example restore command.

PostgreSQL and MySQL

PostgreSQL and MySQL

PostgreSQL and MySQLPostgreSQL Experts, Inc.

This document compares the two major open source databases: MySQL and PostgreSQL. It provides a brief history of each database's development. MySQL prioritized ease-of-use and performance early on, while PostgreSQL focused on features, security, and standards compliance. More recently, both databases have expanded their feature sets. The document discusses the most common uses, features, and performance of each database. It concludes that for simple queries on 2-core machines, MySQL may perform better, while PostgreSQL tends to perform better for complex queries that can leverage multiple CPU cores.

HBase Advanced - Lars George

HBase Advanced - Lars George

HBase Advanced - Lars GeorgeJAX London

PostgreSQL as an Alternative to MSSQL

PostgreSQL as an Alternative to MSSQL

PostgreSQL as an Alternative to MSSQLAlexei Krasner

Researching an alternative to the MS SQL database - first of all in order to gain additional technological benefits, secondly moving towards an open source way of development. The idea behind this presentation was to introduce PostgreSQL (ver. 9.4+) in a different manner than a conventional "Pros Vs. Cons" style, it is more likely to be a "Buzz Word" thesaurus (of course based on a deep research). P.S. Since it's a presentation, there was no intention going over and covering all of the PostgreSQL features - most of the interesting parts.

Some key value stores using log-structure

Some key value stores using log-structure

Some key value stores using log-structureZhichao Liang

Ten Reasons Why You Should Prefer PostgreSQL to MySQL

Ten Reasons Why You Should Prefer PostgreSQL to MySQL

Ten Reasons Why You Should Prefer PostgreSQL to MySQLanandology

Ansible for large scale deployment

Ansible for large scale deployment

Ansible for large scale deploymentKarthik .P.R

When is MyRocks good?

When is MyRocks good?

When is MyRocks good? Alkin Tezuysal

Elastic HBase on Mesos - HBaseCon 2015

Elastic HBase on Mesos - HBaseCon 2015

Elastic HBase on Mesos - HBaseCon 2015Cosmin Lehene

Elastic HBase on Mesos aims to improve resource utilization of HBase clusters by running HBase in Docker containers managed by Mesos and Marathon. This allows HBase clusters to dynamically scale based on varying workload demands, increases utilization by running mixed workloads on shared resources, and simplifies operations through standard containerization. Key benefits include easier management, higher efficiency through elastic scaling and resource sharing, and improved cluster tunability.

Non-Relational Databases & Key/Value Stores

Non-Relational Databases & Key/Value Stores

Non-Relational Databases & Key/Value StoresJoël Perras

Key-Value Stores: a practical overview

Key-Value Stores: a practical overview

Key-Value Stores: a practical overviewMarc Seeger

The document provides an overview of key-value stores and discusses CouchDB, Tokyo Cabinet, Redis, and Cassandra as alternatives to relational databases for web applications. It describes each system's features and interface to the Ruby programming language. CouchDB uses JavaScript for generating views and stores schema-less data accessed via HTTP/JSON. Tokyo Cabinet supports hashtable, B-tree, and table modes and is used by Mixi, a Japanese social network. Redis is an in-memory key-value store focused on performance. Cassandra is a distributed, structured key-value store developed by Facebook based on Amazon's Dynamo database.

Ad

More Related Content

What's hot (20)

Dissecting Scalable Database Architectures

Dissecting Scalable Database Architectures

Dissecting Scalable Database Architectureshypertable

M|18 How to use MyRocks with MariaDB Server

M|18 How to use MyRocks with MariaDB Server

M|18 How to use MyRocks with MariaDB ServerMariaDB plc

MyRocks in MariaDB summarizes MyRocks, a storage engine for MariaDB that is based on RocksDB. It discusses how MyRocks addresses some of the limitations of InnoDB such as high write and space amplification. It provides details on installing and using MyRocks, including data loading techniques, tuning considerations, and replication support. Parallel replication is supported, but the highest isolation level is repeatable-read and row-based replication must be used.

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDBAthiq Ahamed

This document provides a summary of a presentation that benchmarked the performance of three popular NoSQL databases: Apache Cassandra, Apache HBase, and MongoDB. It describes the architectures and data models of each database. Benchmark tests were run using the Yahoo Cloud Serving Benchmark and found that Apache Cassandra consistently outperformed the other databases across different workloads in terms of load time, read and write performance, and latency. The presentation emphasizes the importance of benchmarks for evaluating NoSQL database performance and choosing the right database based on application requirements.

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...slashn

The document summarizes the evolution of Flipkart's website architecture from 2007 to 2012. Key issues addressed included slow website performance due to slow database queries, isolating reads from writes, isolating production traffic from analytics jobs, implementing caching which introduced complexity, isolating the impact of slow external services, handling spikes in traffic, and separating systems to isolate internal from external requests. The evolution involved learning lessons around scaling databases, isolating systems, managing caching complications, and ensuring systems are not overloaded.

微博cache设计谈Tim Y

The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs like followers and following lists. The caches aim to optimize for fast retrieval of recent home timelines and individual user feeds. Mutexes are used to synchronize cache updates between servers.

Digital Library Collection Management using HBase

Digital Library Collection Management using HBase

Digital Library Collection Management using HBaseHBaseCon

Cassandra Introduction & Features

Cassandra Introduction & Features

Cassandra Introduction & FeaturesPhil Peace

Cassandra is an open source, distributed, decentralized, and fault-tolerant NoSQL database that is highly scalable and provides tunable consistency. It was created at Facebook based on Amazon's Dynamo and Google's Bigtable. Cassandra's key features include elastic scalability through horizontal partitioning, high availability with no single point of failure, tunable consistency levels, and a column-oriented data model with a CQL interface. Major companies like eBay, Netflix, and Apple use Cassandra for applications requiring large volumes of writes, geographical distribution, and evolving data models.

Intro to big data choco devday - 23-01-2014

Intro to big data choco devday - 23-01-2014

Intro to big data choco devday - 23-01-2014Hassan Islamov

This document provides an introduction to big data and Hadoop. It discusses the growth of data from 2006 to 2020. It then introduces key concepts of Hadoop including HDFS, MapReduce, and the Hadoop ecosystem. It describes how HDFS stores and processes large datasets in a distributed manner through block storage on datanodes and metadata management by the namenode. MapReduce provides a programming model for distributed processing of large datasets across clusters. The document also discusses challenges of hardware failures and solutions in Hadoop like HDFS high availability and federation.

HBase: Extreme Makeover

HBase: Extreme Makeover

HBase: Extreme MakeoverHBaseCon

Speaker: Vladimir Rodionov (bigbase.org) This talks introduces a totally new implementation of a multilayer caching in HBase called BigBase. BigBase has a big advantage over HBase 0.94/0.96 because of an ability to utilize all available server RAM in the most efficient way, and because of a novel implementation of a L3 level cache on fast SSDs. The talk will show that different type of caches in BigBase work best for different type of workloads, and that a combination of these caches (L1/L2/L3) increases the overall performance of HBase by a very wide margin.

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?Accumulo Summit

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed Storage

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed Storage

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed StorageCloudera, Inc.

Pinterest uses Apache HBase to store data for users' personalized "following feeds" at scale. This involves storing billions of pins and updates per day. Some key challenges addressed are handling high throughput writes from fanouts, providing low latency reads, and resolving potential data inconsistencies from race conditions. Optimizations to HBase include increased memstore size, block cache tuning, and prefix compression. Maintaining high availability involves writing to dual clusters, tight Zookeeper timeouts, and automated repairs.

MyDUMPER : Faster logical backups and restores

MyDUMPER : Faster logical backups and restores

MyDUMPER : Faster logical backups and restores Mydbops

This document discusses using mydumper and myloader tools to perform faster logical backups and restores of MySQL databases. Mydumper is an open source backup tool that can backup databases in parallel threads, split output into multiple files, compress files, and support selective backups of databases and tables. It has advantages over the built-in mysqldump such as being multithreaded, supporting compression, and ability to chunk output. The document provides details on mydumper features, options, and examples of using it for full and partial backups. It also introduces myloader for restoring mydumper backups and provides an example restore command.

PostgreSQL and MySQL

PostgreSQL and MySQL

PostgreSQL and MySQLPostgreSQL Experts, Inc.

This document compares the two major open source databases: MySQL and PostgreSQL. It provides a brief history of each database's development. MySQL prioritized ease-of-use and performance early on, while PostgreSQL focused on features, security, and standards compliance. More recently, both databases have expanded their feature sets. The document discusses the most common uses, features, and performance of each database. It concludes that for simple queries on 2-core machines, MySQL may perform better, while PostgreSQL tends to perform better for complex queries that can leverage multiple CPU cores.

HBase Advanced - Lars George

HBase Advanced - Lars George

HBase Advanced - Lars GeorgeJAX London

PostgreSQL as an Alternative to MSSQL

PostgreSQL as an Alternative to MSSQL

PostgreSQL as an Alternative to MSSQLAlexei Krasner

Researching an alternative to the MS SQL database - first of all in order to gain additional technological benefits, secondly moving towards an open source way of development. The idea behind this presentation was to introduce PostgreSQL (ver. 9.4+) in a different manner than a conventional "Pros Vs. Cons" style, it is more likely to be a "Buzz Word" thesaurus (of course based on a deep research). P.S. Since it's a presentation, there was no intention going over and covering all of the PostgreSQL features - most of the interesting parts.

Some key value stores using log-structure

Some key value stores using log-structure

Some key value stores using log-structureZhichao Liang

Ten Reasons Why You Should Prefer PostgreSQL to MySQL

Ten Reasons Why You Should Prefer PostgreSQL to MySQL

Ten Reasons Why You Should Prefer PostgreSQL to MySQLanandology

Ansible for large scale deployment

Ansible for large scale deployment

Ansible for large scale deploymentKarthik .P.R

When is MyRocks good?

When is MyRocks good?

When is MyRocks good? Alkin Tezuysal

Elastic HBase on Mesos - HBaseCon 2015

Elastic HBase on Mesos - HBaseCon 2015

Elastic HBase on Mesos - HBaseCon 2015Cosmin Lehene

Elastic HBase on Mesos aims to improve resource utilization of HBase clusters by running HBase in Docker containers managed by Mesos and Marathon. This allows HBase clusters to dynamically scale based on varying workload demands, increases utilization by running mixed workloads on shared resources, and simplifies operations through standard containerization. Key benefits include easier management, higher efficiency through elastic scaling and resource sharing, and improved cluster tunability.

Dissecting Scalable Database Architectures

Dissecting Scalable Database Architectures

Dissecting Scalable Database Architectureshypertable

M|18 How to use MyRocks with MariaDB Server

M|18 How to use MyRocks with MariaDB Server

M|18 How to use MyRocks with MariaDB ServerMariaDB plc

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB

Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDBAthiq Ahamed

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...

Slash n: Tech Talk Track 2 – Website Architecture-Mistakes & Learnings - Sidd...slashn

微博cache设计谈Tim Y

Digital Library Collection Management using HBase

Digital Library Collection Management using HBase

Digital Library Collection Management using HBaseHBaseCon

Cassandra Introduction & Features

Cassandra Introduction & Features

Cassandra Introduction & FeaturesPhil Peace

Intro to big data choco devday - 23-01-2014

Intro to big data choco devday - 23-01-2014

Intro to big data choco devday - 23-01-2014Hassan Islamov

HBase: Extreme Makeover

HBase: Extreme Makeover

HBase: Extreme MakeoverHBaseCon

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?Accumulo Summit

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed Storage

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed Storage

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed StorageCloudera, Inc.

MyDUMPER : Faster logical backups and restores

MyDUMPER : Faster logical backups and restores

MyDUMPER : Faster logical backups and restores Mydbops

PostgreSQL and MySQL

PostgreSQL and MySQL

PostgreSQL and MySQLPostgreSQL Experts, Inc.

HBase Advanced - Lars George

HBase Advanced - Lars George

HBase Advanced - Lars GeorgeJAX London

PostgreSQL as an Alternative to MSSQL

PostgreSQL as an Alternative to MSSQL

PostgreSQL as an Alternative to MSSQLAlexei Krasner

Some key value stores using log-structure

Some key value stores using log-structure

Some key value stores using log-structureZhichao Liang

Ten Reasons Why You Should Prefer PostgreSQL to MySQL

Ten Reasons Why You Should Prefer PostgreSQL to MySQL

Ten Reasons Why You Should Prefer PostgreSQL to MySQLanandology

Ansible for large scale deployment

Ansible for large scale deployment

Ansible for large scale deploymentKarthik .P.R

When is MyRocks good?

When is MyRocks good?

When is MyRocks good? Alkin Tezuysal

Elastic HBase on Mesos - HBaseCon 2015

Elastic HBase on Mesos - HBaseCon 2015

Elastic HBase on Mesos - HBaseCon 2015Cosmin Lehene

Viewers also liked (12)

Non-Relational Databases & Key/Value Stores

Non-Relational Databases & Key/Value Stores

Non-Relational Databases & Key/Value StoresJoël Perras

Key-Value Stores: a practical overview

Key-Value Stores: a practical overview

Key-Value Stores: a practical overviewMarc Seeger

The document provides an overview of key-value stores and discusses CouchDB, Tokyo Cabinet, Redis, and Cassandra as alternatives to relational databases for web applications. It describes each system's features and interface to the Ruby programming language. CouchDB uses JavaScript for generating views and stores schema-less data accessed via HTTP/JSON. Tokyo Cabinet supports hashtable, B-tree, and table modes and is used by Mixi, a Japanese social network. Redis is an in-memory key-value store focused on performance. Cassandra is a distributed, structured key-value store developed by Facebook based on Amazon's Dynamo database.

Informix NoSQL & Hybrid SQL detailed deep dive

Informix NoSQL & Hybrid SQL detailed deep dive

Informix NoSQL & Hybrid SQL detailed deep diveKeshav Murthy

This document provides an overview of Informix NoSQL capabilities and use cases. It discusses key-value stores, column family stores, document databases, and graph databases supported by Informix NoSQL. Several business uses of Informix NoSQL are outlined, including session store, user profile store, content metadata store, mobile apps, third party data aggregation, caching, and ecommerce. The document also compares pricing of Informix and MongoDB editions over a three year period. Finally, it provides timelines for go-to-market strategies for DB2 JSON and Informix JSON capabilities.

Column Stores and Google BigQuery

Column Stores and Google BigQuery

Column Stores and Google BigQueryCsaba Toth

NoSQL support in Informix (JSON storage, Mongo DB API)

NoSQL support in Informix (JSON storage, Mongo DB API)

NoSQL support in Informix (JSON storage, Mongo DB API)Keshav Murthy

Key-Value Pairs

Key-Value Pairs

Key-Value Pairslittledata

Key-value pairs are a fundamental concept in databases where a key is a field name or identifier and the value is the data stored in that field. For example, in an address book database each contact record would contain a "city" field with the key and the city name like "Buffalo" or "Toronto" as the value. Consistency in database design means each record contains the same keys so queries can easily search values by key like finding all contacts living in a particular city.

VLDB 2009 Tutorial on Column-Stores

VLDB 2009 Tutorial on Column-Stores

VLDB 2009 Tutorial on Column-StoresDaniel Abadi

This document summarizes a tutorial on column-oriented database systems given at VLDB 2009. It discusses the evolution of column-oriented databases from early systems like DSM to current commercial systems. Key features of column-oriented databases include storing data by column rather than by row for improved read efficiency, better compression, and indexing capabilities. The tutorial outlines optimizations made possible by the columnar format like late materialization and vectorized processing. It also compares the performance of column and row storage using a telco data warehousing example.

NoSQL Database: Classification, Characteristics and Comparison

NoSQL Database: Classification, Characteristics and Comparison

NoSQL Database: Classification, Characteristics and ComparisonMayuree Srikulwong

Big data key-value and column stores redis - cassandra

Big data key-value and column stores redis - cassandra

Big data key-value and column stores redis - cassandraJWORKS powered by Ordina

The document provides an introduction to NoSQL database types, including key/value stores, column stores, document stores, and graph databases. It discusses the strengths and weaknesses of each type as well as common use cases. It also covers the CAP theorem, which states that a distributed system cannot simultaneously provide consistency, availability, and partition tolerance. The document establishes that different NoSQL database types adhere to different parts of the CAP theorem and provides examples for each. Setup instructions are also included to access code repositories for hands-on exercises with Cassandra and Redis.

Redis persistence in practice

Redis persistence in practice

Redis persistence in practiceEugene Fidelin

MongoDB Introduction - Document Oriented Nosql Database

MongoDB Introduction - Document Oriented Nosql Database

MongoDB Introduction - Document Oriented Nosql DatabaseSudhir Patil

- MongoDB is a document-oriented, non-relational database that uses JSON-like documents with dynamic schemas, horizontal scaling with sharding and replication, and high performance. - It is used by many large companies and is a good fit for huge, unstructured data or when dynamic schemas are needed. However, it does not support transactions or have the same data integrity features as relational databases. - MongoDB can store data in flexible document formats, allows ad hoc queries, and supports common operations like insert, update, delete. Features include replication, sharding, indexing, geospatial queries and MapReduce.

Intro to column stores

Intro to column stores

Intro to column storesJustin Swanhart

Non-Relational Databases & Key/Value Stores

Non-Relational Databases & Key/Value Stores

Non-Relational Databases & Key/Value StoresJoël Perras

Key-Value Stores: a practical overview

Key-Value Stores: a practical overview

Key-Value Stores: a practical overviewMarc Seeger

Informix NoSQL & Hybrid SQL detailed deep dive

Informix NoSQL & Hybrid SQL detailed deep dive

Informix NoSQL & Hybrid SQL detailed deep diveKeshav Murthy

Column Stores and Google BigQuery

Column Stores and Google BigQuery

Column Stores and Google BigQueryCsaba Toth

NoSQL support in Informix (JSON storage, Mongo DB API)

NoSQL support in Informix (JSON storage, Mongo DB API)

NoSQL support in Informix (JSON storage, Mongo DB API)Keshav Murthy

Key-Value Pairs

Key-Value Pairs

Key-Value Pairslittledata

VLDB 2009 Tutorial on Column-Stores

VLDB 2009 Tutorial on Column-Stores

VLDB 2009 Tutorial on Column-StoresDaniel Abadi

NoSQL Database: Classification, Characteristics and Comparison

NoSQL Database: Classification, Characteristics and Comparison

NoSQL Database: Classification, Characteristics and ComparisonMayuree Srikulwong

Big data key-value and column stores redis - cassandra

Big data key-value and column stores redis - cassandra

Big data key-value and column stores redis - cassandraJWORKS powered by Ordina

Redis persistence in practice

Redis persistence in practice

Redis persistence in practiceEugene Fidelin

MongoDB Introduction - Document Oriented Nosql Database

MongoDB Introduction - Document Oriented Nosql Database

MongoDB Introduction - Document Oriented Nosql DatabaseSudhir Patil

Intro to column stores

Intro to column stores

Intro to column storesJustin Swanhart

Ad

Similar to Key-Value-Stores -- The Key to Scaling? (20)

Memory: The New Disk

Memory: The New Disk

Memory: The New DiskTim Lossen

This document discusses using Redis as a database for the backend of a Facebook game application. It describes the requirements of supporting 1 million daily users with high write throughput needs. A Redis database was chosen because it provides fast in-memory performance suitable for the application's random access workload. Redis was able to meet the throughput requirements of 200,000 requests per minute and support storing 100KB of data per user in memory. The document provides advice to choose the right tool for the job and avoid sharding until necessary to keep the database configuration simple.

Drop acidMike Feltman

Wmware NoSQLMurat Çakal

This document provides an introduction to using Spring Data to simplify development of NoSQL applications. It discusses why NoSQL databases emerged as alternatives to relational databases, gives an overview of popular NoSQL databases like Redis, MongoDB, Neo4j and their features. It then introduces Spring Data and how it provides common APIs and conventions to work with various NoSQL databases. Specific database APIs for MongoDB, HyperSQL and Neo4j are also covered along with how Spring Data supports cross-store persistence across SQL and NoSQL databases in a single transaction.

KeyValue Stores

KeyValue Stores

KeyValue StoresMauro Pompilio

Technical overview of three of the most representative KeyValue Stores: Cassandra, Redis and CouchDB. Focused on Ruby and Ruby on Rails developement, this talk shows how to solve common problems, the most popular libraries, benchmarking and the best use case for each one of them. This talk was part of the Conferencia Rails 2009, Madrid, Spain. http://app.conferenciarails.org/talks/43-key-value-stores-conviertete-en-un-jedi-master

Large-scale Web Apps @ Pinterest

Large-scale Web Apps @ Pinterest

Large-scale Web Apps @ PinterestHBaseCon

Speaker: Varun Sharma (Pinterest) Over the past year, HBase has become an integral component of Pinterest's storage stack. HBase has enabled us to quickly launch and iterate on new products and create amazing pinner experiences. This talk briefly describes some of these applications, the underlying schema, and how our HBase setup stays highly available and performant despite billions of requests every week. It will also include some performance tips for running on SSDs. Finally, we will talk about a homegrown serving technology we built from a mashup of HBase components that has gained wide adoption across Pinterest.

C* Summit 2013: Cassandra at eBay Scale by Feng Qu and Anurag Jambhekar

C* Summit 2013: Cassandra at eBay Scale by Feng Qu and Anurag Jambhekar

C* Summit 2013: Cassandra at eBay Scale by Feng Qu and Anurag JambhekarDataStax Academy

We have seen rapid adoption of C* at eBay in past two years. We have made tremendous efforts to integrate C* into existing database platforms, including Oracle, MySQL, Postgres, MongoDB, XMP etc.. We also scale C* to meet business requirement and encountered technical challenges you only see at eBay scale, 100TB data on hundreds of nodes. We will share our experience of deployment automation, managing, monitoring, reporting for both Apache Cassandra and DataStax enterprise.

Petabyte scale on commodity infrastructure

Petabyte scale on commodity infrastructure

Petabyte scale on commodity infrastructureelliando dias

This document discusses Hadoop, an open-source software framework for distributed storage and processing of large datasets across clusters of commodity servers. It describes how Hadoop addresses the need to reliably process huge datasets using a distributed file system and MapReduce processing on commodity hardware. It also provides details on how Hadoop has been implemented and used at Yahoo to process petabytes of data and support thousands of jobs weekly on large clusters.

Cassandra at Vast

Cassandra at Vast

Cassandra at VastCassandra Austin

Austin Cassandra Users 6/19: Apache Cassandra at Vast

Austin Cassandra Users 6/19: Apache Cassandra at Vast

Austin Cassandra Users 6/19: Apache Cassandra at VastDataStax Academy

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)Bob Pusateri

My Sql And Search At Craigslist

My Sql And Search At Craigslist

My Sql And Search At CraigslistMySQLConference

This document summarizes Jeremy Zawodny's work with MySQL and search at Craigslist. It discusses how Craigslist uses MySQL for its classified listings but encountered scaling issues as traffic grew. To address this, Craigslist implemented the Sphinx search engine, which improved performance and allowed them to reduce their MySQL cluster size. The document also outlines Craigslist's data archiving strategy using eventual consistency and their goals for further optimizing their database and search infrastructure.

Using Spring with NoSQL databases (SpringOne China 2012)

Using Spring with NoSQL databases (SpringOne China 2012)

Using Spring with NoSQL databases (SpringOne China 2012)Chris Richardson

Introduction to Hadoop and Big Data

Introduction to Hadoop and Big Data

Introduction to Hadoop and Big DataJoe Alex

This document provides an introduction to Hadoop and big data. It discusses the new kinds of large, diverse data being generated and the need for platforms like Hadoop to process and analyze this data. It describes the core components of Hadoop, including HDFS for distributed storage and MapReduce for distributed processing. It also discusses some of the common applications of Hadoop and other projects in the Hadoop ecosystem like Hive, Pig, and HBase that build on the core Hadoop framework.

Webinar - DreamObjects/Ceph Case Study

Webinar - DreamObjects/Ceph Case Study

Webinar - DreamObjects/Ceph Case StudyCeph Community

This document summarizes DreamObjects, an object storage platform powered by Ceph. It discusses the hardware used in storage and support nodes, including Intel and AMD processors, RAM, disks, and networking components. The document also provides details on Ceph configuration including replication, CRUSH mapping, OSD configuration, and application tuning. Monitoring tools discussed include Chef, pdsh, Sensu, collectd, graphite, logstash, Jenkins and future plans.

M6d cassandrapresentation

M6d cassandrapresentation

M6d cassandrapresentationEdward Capriolo

Agility and Scalability with MongoDB

Agility and Scalability with MongoDB

Agility and Scalability with MongoDBMongoDB

MongoDB has taken a clear lead in adoption among the new generation of databases, including the enormous variety of NoSQL offerings. A key reason for this lead has been a unique combination of agility and scalability. Agility provides business units with a quick start and flexibility to maintain development velocity, despite changing data and requirements. Scalability maintains that flexibility while providing fast, interactive performance as data volume and usage increase. We'll address the key organizational, operational, and engineering considerations to ensure that agility and scalability stay aligned at increasing scale, from small development instances to web-scale applications. We will also survey some key examples of highly-scaled customer applications of MongoDB.

Hadoop-Quick introduction

Hadoop-Quick introduction

Hadoop-Quick introductionSandeep Singh

- Data is a precious resource that can last longer than the systems themselves (Tim Berners-Lee) - Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It provides reliability, scalability and flexibility. - Hadoop consists of HDFS for storage and MapReduce for processing. The main nodes include NameNode, DataNodes, JobTracker and TaskTrackers. Tools like Hive, Pig, HBase extend its capabilities for SQL-like queries, data flows and NoSQL access.

Top 10 lessons learned from deploying hadoop in a private cloud

Top 10 lessons learned from deploying hadoop in a private cloud

Top 10 lessons learned from deploying hadoop in a private cloudRogue Wave Software

This document discusses lessons learned from deploying Hadoop in a private cloud. Some key lessons include: choosing the right hardware with sufficient CPU, RAM, and bandwidth; understanding that configuration is critical for Hadoop, HBase, and Solr; expecting failures and bugs to occur; and realizing that big data projects take a long time to complete. Public clouds are expensive for long-term big data storage needs, so a private cloud may be more cost effective despite requiring infrastructure management. Open source tools like Hadoop have advanced to enable organizations to tackle "big data" challenges.

Scaling the Web: Databases & NoSQL

Scaling the Web: Databases & NoSQL

Scaling the Web: Databases & NoSQLRichard Schneeman

This is an introduction to relational and non-relational databases and how their performance affects scaling a web application. This is a recording of a guest Lecture I gave at the University of Texas school of Information. In this talk I address the technologies and tools Gowalla (gowalla.com) uses including memcache, redis and cassandra. Find more on my blog: http://schneems.com

Redis e Memcached - Daniel Naves - Omnilogic

Redis e Memcached - Daniel Naves - Omnilogic

Redis e Memcached - Daniel Naves - OmnilogicFelipe Guimarães

Redis and Memcached are both open-source, in-memory key-value data structures stores that are commonly used for caching, but Redis has additional features like persistence, data structures, and pub/sub capabilities that make it more flexible than the simpler Memcached. Real-world use cases for Redis include caching page fragments to speed up websites by 5x, job queuing with persistence and multi-queue/worker support, and caching model predictions to speed up machine learning workflows by 100x.

Memory: The New Disk

Memory: The New Disk

Memory: The New DiskTim Lossen

Drop acidMike Feltman

Wmware NoSQLMurat Çakal

KeyValue Stores

KeyValue Stores

KeyValue StoresMauro Pompilio

Large-scale Web Apps @ Pinterest

Large-scale Web Apps @ Pinterest

Large-scale Web Apps @ PinterestHBaseCon

C* Summit 2013: Cassandra at eBay Scale by Feng Qu and Anurag Jambhekar

C* Summit 2013: Cassandra at eBay Scale by Feng Qu and Anurag Jambhekar

C* Summit 2013: Cassandra at eBay Scale by Feng Qu and Anurag JambhekarDataStax Academy

Petabyte scale on commodity infrastructure

Petabyte scale on commodity infrastructure

Petabyte scale on commodity infrastructureelliando dias

Cassandra at Vast

Cassandra at Vast

Cassandra at VastCassandra Austin

Austin Cassandra Users 6/19: Apache Cassandra at Vast

Austin Cassandra Users 6/19: Apache Cassandra at Vast

Austin Cassandra Users 6/19: Apache Cassandra at VastDataStax Academy

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)

Select Stars: A DBA's Guide to Azure Cosmos DB (SQL Saturday Oslo 2018)Bob Pusateri

My Sql And Search At Craigslist

My Sql And Search At Craigslist

My Sql And Search At CraigslistMySQLConference

Using Spring with NoSQL databases (SpringOne China 2012)

Using Spring with NoSQL databases (SpringOne China 2012)

Using Spring with NoSQL databases (SpringOne China 2012)Chris Richardson

Introduction to Hadoop and Big Data

Introduction to Hadoop and Big Data

Introduction to Hadoop and Big DataJoe Alex

Webinar - DreamObjects/Ceph Case Study

Webinar - DreamObjects/Ceph Case Study

Webinar - DreamObjects/Ceph Case StudyCeph Community

M6d cassandrapresentation

M6d cassandrapresentation

M6d cassandrapresentationEdward Capriolo

Agility and Scalability with MongoDB

Agility and Scalability with MongoDB

Agility and Scalability with MongoDBMongoDB

Hadoop-Quick introduction

Hadoop-Quick introduction

Hadoop-Quick introductionSandeep Singh

Top 10 lessons learned from deploying hadoop in a private cloud

Top 10 lessons learned from deploying hadoop in a private cloud

Top 10 lessons learned from deploying hadoop in a private cloudRogue Wave Software

Scaling the Web: Databases & NoSQL

Scaling the Web: Databases & NoSQL

Scaling the Web: Databases & NoSQLRichard Schneeman

Redis e Memcached - Daniel Naves - Omnilogic

Redis e Memcached - Daniel Naves - Omnilogic

Redis e Memcached - Daniel Naves - OmnilogicFelipe Guimarães

Ad

More from Tim Lossen (20)

Apocalypse Later

Apocalypse Later

Apocalypse LaterTim Lossen

This document discusses the fragility of civilization and importance of preparing for long-term survival. It contains several bullet points stating that civilization is more fragile than it seems, survival is a long-term game, and now is the best time to start preparing. There are also images showing technological progress over time and links related to survival skills and seed banking. The overall message is that civilization could collapse and individuals should start developing skills to be self-sufficient.

Embracing Failure

Embracing Failure

Embracing FailureTim Lossen

Embracing failure is important for game developers according to Tim Lossen of Wooga. They use a portfolio approach, developing multiple games simultaneously so a failure of one game does not jeopardize the company. It is also important to fail early in the development process through frequent testing. While the emotional side of failure is difficult, being proud of failures can help companies improve through learning.

Embracing Failure

Embracing Failure

Embracing FailureTim Lossen

Embracing Failure is a document about why and how game company Wooga embraces failure in their development process. It discusses how Wooga uses a "hit filter" to identify promising games from a portfolio of projects, reviews ideas early, and cultivates a culture where it's okay to fail quickly. The document advocates embracing failure through a portfolio approach, making hard decisions, and viewing failures as learning opportunities rather than punishments in order to encourage risk-taking and innovation.

Enemy of the State?

Enemy of the State?

Enemy of the State?Tim Lossen

Beyond DevopsTim Lossen

The document discusses 12 patterns for collaboration beyond DevOps at a company called Wooga. The patterns are divided into three categories: Broadcasting, Self Service, and Connecting. Broadcasting patterns include weekly info meetings and brownbag sessions. Self Service patterns involve open data and internal directories. Connecting patterns promote mentorship, lunches, and social events to foster relationships among employees.

All Your Core Are Belong To Us

All Your Core Are Belong To Us

All Your Core Are Belong To UsTim Lossen

Podularity FTW!

Podularity FTW!

Podularity FTW!Tim Lossen

Cubic Foot Gardening

Cubic Foot Gardening

Cubic Foot GardeningTim Lossen

Dashboard Mania

Dashboard Mania

Dashboard ManiaTim Lossen

The document discusses using dashboards and information radiators to track key metrics and provide visibility. It provides examples of metrics that could be tracked, such as requests per second and installs per minute. It also discusses Kafka, a tool for building real-time data pipelines and streaming applications, and how it provides high-throughput, persistent, publish-subscribe messaging capabilities. The document recommends not flying blind, checking out Kafka, and being creative with dashboards and information radiators.

Event-Stream Processing with Kafka

Event-Stream Processing with Kafka

Event-Stream Processing with KafkaTim Lossen

Kafka is a high-throughput, persistent, and publish-subscribe messaging system. It uses a producer-broker-consumer model where producers publish messages to topics, brokers distribute messages to consumers, and consumers subscribe to topics. The document discusses how game company Wooga uses Kafka for event streaming to track metrics like requests per second and installs per minute. It also describes Wooga's open source Kafka libraries for Ruby and Erlang called Kafkaesque.

Into the VoidTim Lossen

Tim Lossen discusses the challenges of building social games at Wooga including high traffic of 2 million users and 15 million visits per day, and storing large amounts of user data for their 40 million total users. He notes that hot game data needs fast access and questions whether a database is even needed. Lossen also provides recommendations on using Amazon S3 for storage, the JRuby programming language, and challenging assumptions in their systems.

Treasure Island -- Concurrency in JRuby

Treasure Island -- Concurrency in JRuby

Treasure Island -- Concurrency in JRubyTim Lossen

The document discusses various approaches to concurrency in programming languages like Ruby, Java, Erlang and Scala. It covers different implementations of concurrency in Ruby like threads, actors, and software transactional memory. The document also mentions several frameworks for concurrency including Akka, Disruptor, and evented architectures and includes links to additional resources on the topic.

Dashboard Mania

Dashboard Mania

Dashboard ManiaTim Lossen

JRuby vs. Rubinius

JRuby vs. Rubinius

JRuby vs. RubiniusTim Lossen

Redis to the Rescue?

Redis to the Rescue?

Redis to the Rescue?Tim Lossen

The document discusses how the mobile game company wooga used Redis as the data store for their Facebook game Happy Hospital to achieve their goals of 1 million daily active users with a small production cluster. It describes their data model of storing user and game entity data in Redis hashes and lists, and how their architecture evolved over time from the initial setup in 2010 to the current production setup in 2011 to address Redis limitations around memory usage and durability. The document highlights Redis' performance and stability but also warns of memory consumption and durability issues.

Redis to the Rescue?

Redis to the Rescue?

Redis to the Rescue?Tim Lossen

Redis was used to improve performance for two game companies experiencing rapid growth. For Monster World, moving write-intensive inventory data to Redis reduced load on MySQL by 10% and allowed further data migration. For Happy Hospital, Redis served as the main database to handle peak loads of 4000 requests/second, achieving response times of 10ms. The main challenge was ensuring durability with Redis' virtual memory, but a diskstore solution currently in development was promising. Both cases found Redis provided order-of-magnitude performance improvements over MySQL.

The Smallest Cluster in the World

The Smallest Cluster in the World

The Smallest Cluster in the WorldTim Lossen

The document discusses the GA144, the smallest computer cluster in the world. It is a single-chip system with 144 processing nodes, each with 20KB of memory, capable of 96 billion operations per second while consuming only 14-650 milliwatts of power. It provides parallel processing capabilities on a small scale and is being used for applications like cryptography, signal processing, and neural networks.

Announcing Euruko 2011

Announcing Euruko 2011

Announcing Euruko 2011Tim Lossen

An Introduction to Membase

An Introduction to Membase

An Introduction to MembaseTim Lossen

Petascale Storage -- Do It Yourself!

Petascale Storage -- Do It Yourself!

Petascale Storage -- Do It Yourself!Tim Lossen

Apocalypse Later

Apocalypse Later

Apocalypse LaterTim Lossen

Embracing Failure

Embracing Failure

Embracing FailureTim Lossen

Embracing Failure

Embracing Failure

Embracing FailureTim Lossen

Enemy of the State?

Enemy of the State?

Enemy of the State?Tim Lossen

Beyond DevopsTim Lossen

All Your Core Are Belong To Us

All Your Core Are Belong To Us

All Your Core Are Belong To UsTim Lossen

Podularity FTW!

Podularity FTW!

Podularity FTW!Tim Lossen

Cubic Foot Gardening

Cubic Foot Gardening

Cubic Foot GardeningTim Lossen

Dashboard Mania

Dashboard Mania

Dashboard ManiaTim Lossen

Event-Stream Processing with Kafka

Event-Stream Processing with Kafka

Event-Stream Processing with KafkaTim Lossen

Into the VoidTim Lossen

Treasure Island -- Concurrency in JRuby

Treasure Island -- Concurrency in JRuby

Treasure Island -- Concurrency in JRubyTim Lossen

Dashboard Mania

Dashboard Mania

Dashboard ManiaTim Lossen

JRuby vs. Rubinius

JRuby vs. Rubinius

JRuby vs. RubiniusTim Lossen

Redis to the Rescue?

Redis to the Rescue?

Redis to the Rescue?Tim Lossen

Redis to the Rescue?

Redis to the Rescue?

Redis to the Rescue?Tim Lossen

The Smallest Cluster in the World

The Smallest Cluster in the World

The Smallest Cluster in the WorldTim Lossen

Announcing Euruko 2011

Announcing Euruko 2011

Announcing Euruko 2011Tim Lossen

An Introduction to Membase

An Introduction to Membase

An Introduction to MembaseTim Lossen

Petascale Storage -- Do It Yourself!

Petascale Storage -- Do It Yourself!

Petascale Storage -- Do It Yourself!Tim Lossen

Recently uploaded (20)

Semantic Cultivators : The Critical Future Role to Enable AI

Semantic Cultivators : The Critical Future Role to Enable AI

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

What is Model Context Protocol(MCP) - The new technology for communication bw...

What is Model Context Protocol(MCP) - The new technology for communication bw...

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

How analogue intelligence complements AI

How analogue intelligence complements AI

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

Linux Professional Institute LPIC-1 Exam.pdf

Linux Professional Institute LPIC-1 Exam.pdf

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Linux Support for SMARC: How Toradex Empowers Embedded Developers

Linux Support for SMARC: How Toradex Empowers Embedded Developers

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

Toradex brings robust Linux support to SMARC (Smart Mobility Architecture), ensuring high performance and long-term reliability for embedded applications. Here’s how: • Optimized Torizon OS & Yocto Support – Toradex provides Torizon OS, a Debian-based easy-to-use platform, and Yocto BSPs for customized Linux images on SMARC modules. • Seamless Integration with i.MX 8M Plus and i.MX 95 – Toradex SMARC solutions leverage NXP’s i.MX 8 M Plus and i.MX 95 SoCs, delivering power efficiency and AI-ready performance. • Secure and Reliable – With Secure Boot, over-the-air (OTA) updates, and LTS kernel support, Toradex ensures industrial-grade security and longevity. • Containerized Workflows for AI & IoT – Support for Docker, ROS, and real-time Linux enables scalable AI, ML, and IoT applications. • Strong Ecosystem & Developer Support – Toradex offers comprehensive documentation, developer tools, and dedicated support, accelerating time-to-market. With Toradex’s Linux support for SMARC, developers get a scalable, secure, and high-performance solution for industrial, medical, and AI-driven applications. Do you have a specific project or application in mind where you're considering SMARC? We can help with Free Compatibility Check and help you with quick time-to-market For more information: https://www.toradex.com/computer-on-modules/smarc-arm-family

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

Electronic_Mail_Attacks-1-35.pdf by xploit

Electronic_Mail_Attacks-1-35.pdf by xploit

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Generative Artificial Intelligence (GenAI) in Business

Generative Artificial Intelligence (GenAI) in Business

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Special Meetup Edition - TDX Bengaluru Meetup #52.pptx

Special Meetup Edition - TDX Bengaluru Meetup #52.pptx

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Technology Trends in 2025: AI and Big Data Analytics

Technology Trends in 2025: AI and Big Data Analytics

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including: -Artificial Intelligence Market Overview -Strategies for AI Adoption in 2025 -Anticipated drivers of AI adoption and transformative technologies -Benefits of AI and Big data for your business -Tips on how to prepare your business for innovation -AI and data privacy: Strategies for securing data privacy in AI models, etc. Download your free copy nowand implement the key findings to improve your business.

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

TrsLabs - Fintech Product & Business Consulting

TrsLabs - Fintech Product & Business Consulting

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Want to learn practical tips for designing systems that can scale efficiently without compromising speed? Join us for a workshop where we’ll address these challenges head-on and explore how to architect low-latency systems using Rust. During this free interactive workshop oriented for developers, engineers, and architects, we’ll cover how Rust’s unique language features and the Tokio async runtime enable high-performance application development. As you explore key principles of designing low-latency systems with Rust, you will learn how to: - Create and compile a real-world app with Rust - Connect the application to ScyllaDB (NoSQL data store) - Negotiate tradeoffs related to data modeling and querying - Manage and monitor the database for consistently low latencies

Procurement Insights Cost To Value Guide.pptx

Procurement Insights Cost To Value Guide.pptx

Procurement Insights Cost To Value Guide.pptxJon Hansen

Dev Dives: Automate and orchestrate your processes with UiPath Maestro

Dev Dives: Automate and orchestrate your processes with UiPath Maestro

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Splunk Security Update | Public Sector Summit Germany 2025

Splunk Security Update | Public Sector Summit Germany 2025

Splunk Security Update | Public Sector Summit Germany 2025Splunk

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Semantic Cultivators : The Critical Future Role to Enable AI

Semantic Cultivators : The Critical Future Role to Enable AI

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

What is Model Context Protocol(MCP) - The new technology for communication bw...

What is Model Context Protocol(MCP) - The new technology for communication bw...

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

How analogue intelligence complements AI

How analogue intelligence complements AI

How analogue intelligence complements AIPaul Rowe

Linux Professional Institute LPIC-1 Exam.pdf

Linux Professional Institute LPIC-1 Exam.pdf

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungen

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Linux Support for SMARC: How Toradex Empowers Embedded Developers

Linux Support for SMARC: How Toradex Empowers Embedded Developers

Linux Support for SMARC: How Toradex Empowers Embedded DevelopersToradex

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptx

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

Electronic_Mail_Attacks-1-35.pdf by xploit

Electronic_Mail_Attacks-1-35.pdf by xploit

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Generative Artificial Intelligence (GenAI) in Business

Generative Artificial Intelligence (GenAI) in Business

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

Special Meetup Edition - TDX Bengaluru Meetup #52.pptx

Special Meetup Edition - TDX Bengaluru Meetup #52.pptx

Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55

Technology Trends in 2025: AI and Big Data Analytics

Technology Trends in 2025: AI and Big Data Analytics

Technology Trends in 2025: AI and Big Data AnalyticsInData Labs

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

TrsLabs - Fintech Product & Business Consulting

TrsLabs - Fintech Product & Business Consulting

TrsLabs - Fintech Product & Business ConsultingTrs Labs

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdf

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep Dive

Designing Low-Latency Systems with Rust and ScyllaDB: An Architectural Deep DiveScyllaDB

Procurement Insights Cost To Value Guide.pptx

Procurement Insights Cost To Value Guide.pptx

Procurement Insights Cost To Value Guide.pptxJon Hansen

Dev Dives: Automate and orchestrate your processes with UiPath Maestro

Dev Dives: Automate and orchestrate your processes with UiPath Maestro

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Splunk Security Update | Public Sector Summit Germany 2025

Splunk Security Update | Public Sector Summit Germany 2025

Splunk Security Update | Public Sector Summit Germany 2025Splunk

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

Key-Value-Stores -- The Key to Scaling?

1. Key-Value-Stores: The Key to Scaling? Tim Lossen

2. Who? • @tlossen • backend developer - Ruby, Rails, Sinatra ... • passionate about technology

6. Challenge • backend for facebook game

7. Challenge • backend for facebook game • expected load: - 1 mio. daily active users - 20 mio. total users - 100 KB data per user

8. Challenge • expected peak traﬃc: - 10.000 concurrent users - 200.000 requests / minute

9. Challenge • expected peak traﬃc: - 10.000 concurrent users - 200.000 requests / minute • write-heavy workload

10. Wanted • scalable database • with high throughput - especially for writes

11. Options • relational database - with sharding

12. Options • relational database - with sharding • nosql database - key-value-store - document db - graph db

13. Options • relational database - with sharding • nosql database - key-value-store - document db - graph db

14. Options • relational database - with sharding • nosql database - key-value-store - document db - graph db

15. Options • relational database - with sharding • nosql database - key-value-store - document db - graph db

16. Shortlist • Cassandra • Redis • Membase

18. Facts • written in Java - 55.000 lines of code • Thrift API - clients for Java, Ruby, Python ...

19. History • originally developed by Facebook - in production for “Inbox Search” • later open-sourced - top-level Apache project

20. Features • high availability - no single point of failure • incremental scalability • eventual consistency

21. Architecture • Dynamo-like hash ring - partitioning + replication - all nodes are equal

23. Architecture • Dynamo-like hash ring - partitioning + replication - all nodes are equal • Bigtable data model - column families - supercolumns

25. “Cassandra aims to run on an in"astructure of hundreds of nodes.”

27. Facts • written in C - 13.000 lines of code • socket API - redis-cli - client libs for all major languages

28. Features • high read & write throughput - 50.000 to 100.000 ops / second

29. Features • high read & write throughput - 50.000 to 100.000 ops / second • interesting data structures - lists, hashes, (sorted) sets - atomic operations

30. Features • high read & write throughput - 50.000 to 100.000 ops / second • interesting data structures - lists, hashes, (sorted) sets - atomic operations • strong consistency

31. Architecture • in-memory database - append-only log on disk - virtual memory

32. Architecture • in-memory database - append-only log on disk - virtual memory • single instance - master-slave replication - clustering is on roadmap

33. “Memory is the new disk, disk is the new tape.” — Jim Gray

35. Facts • written in C and Erlang • API-compatible to Memcached - same protocol • client libs for all major languages

36. History • developed by NorthScale & Zynga - used in production (Farmville) • released in June 2010 - Apache 2.0 License

37. Features • “Memcached with persistence” - extremely fast - throughput scales linearly

38. Features • “Memcached with persistence” - extremely fast - throughput scales linearly • automatic data placement - memory, ssd, disk

39. Features • “Memcached with persistence” - extremely fast - throughput scales linearly • automatic data placement - memory, ssd, disk • conﬁgurable replica count

40. Architecture • cluster - all nodes are alike - one elected as “coordinator”

41. Architecture • cluster - all nodes are alike - one elected as “coordinator” • each node is master for part of key space - handles all reads & writes

42. Mapping Scheme

43. “simple, fast, elastic”

45. Which one would you pick?

46. Decision • Cassandra ?

47. Decision • Cassandra ? - too big, too complicated

48. Decision • Cassandra ? - too big, too complicated • Membase ?

49. Decision • Cassandra ? - too big, too complicated • Membase ? - not yet available (then)

50. Decision • Cassandra ? - too big, too complicated • Membase ? - not yet available (then) • Redis !

51. Motivation • keep operations simple • use as few machines as possible - ideally, only one

52. Design • two machines (+ load balancer) - Redis master handles all reads / writes - Redis slave as hot standby

53. Design • two machines (+ load balancer) - Redis master handles all reads / writes - Redis slave as hot standby - both machines used as app servers

54. Design • two machines (+ load balancer) - Redis master handles all reads / writes - Redis slave as hot standby - both machines used as app servers • dedicated hardware

55. Data model • one Redis hash per user - key: facebook id • store data as serialized JSON - booleans, strings, numbers, timestamps ...

56. Advantages • turns Redis into “document db” - eﬃcient to swap user data in / out - atomic ops on parts • easy to dump / restore user data

57. Capacity • 4 GB memory for 20 mio. integer keys - keys always stay in memory!

58. Capacity • 4 GB memory for 20 mio. integer keys - keys always stay in memory! • 2 GB memory for 10.000 user hashes - others can be swapped out

59. Capacity • 4 GB memory for 20 mio. integer keys - keys always stay in memory! • 2 GB memory for 10.000 user hashes - others can be swapped out • 3.6 mio. ops / minute - suﬃcient for 200.000 requests

60. Status • game was launched in august - currently still in beta

61. Status • game was launched in august - currently still in beta • expect to reach 1 mio. daily active users in Q1/2011

62. Status • game was launched in august - currently still in beta • expect to reach 1 mio. daily active users in Q1/2011 • will try to stick to 2 or 3 machines - possibly bigger / faster ones

63. Conclusions • use the right tool for the job

64. Conclusions • use the right tool for the job • keep it simple - avoid sharding, if possible

65. Conclusions • use the right tool for the job • keep it simple - avoid sharding, if possible • don’t scale out too early - but have a viable “plan b”

66. Conclusions • use the right tool for the job • keep it simple - avoid sharding, if possible • don’t scale out too early - but have a viable “plan b” • use dedicated hardware

68. Links • cassandra.apache.org • redis.io • membase.org • tim.lossen.de