Using Redis at Facebook

Mar 30, 20167 likes2,785 views

The document discusses how Redis is used at Facebook to solve two problems: 1) Serving user configurations in real-time by using a hierarchical master-slave replication architecture with global and regional masters and slaves to distribute reads and allow writes at different levels. 2) Processing stats data in real-time by using a split data processing model where daily bulk processing is done separately from real-time processing by tailers that insert stats into a sharded Redis cluster using HINCRBY.

Redis @ Facebook
Guy Yonish
Production Engineering Manager

• Need to read user conﬁgs in real time for every request
• Need to update user conﬁg in some cases
• High request rate - tens of thousands per second. Some
distribution is needed.
• Requests are blocked on the results. Answers are needed
as quickly as possible
Problem #1 - User conﬁguration

Hierarchical master/slave replication
Global Master
Regional Master Regional Master
Slave Slave
Slave
Slave Slave
Slave

Hierarchical master/slave replication
• User conﬁgs are saved as Hashes
• The multi-level replication brings  
the data close to the server
• Reads are distributed across the Slaves.
• If more rps is needed - more slaves can be added - very
scalable.
• Writes can be performed at every level depending on the
need

# Use my fqdn as the shard key
shard_key = node[‘fqdn’]
# Returns a random position in the redis servers array
redis_shard = Digest::MD5.hexdigest(shard_key)
[0…7].to_i(16) % redis_servers.length
my_redis = redis_servers[redis_shard]
Balance the load easily - hash
Easy to use in your attributes ﬁle for every server

Problem #2 - Data processing (stats)
• Data processing and insertion
into long term storage takes
time
• Stats are needed in real time
for the user

Split data processing model
• Perform daily bulk processing of that day. Can take long time.
• Tailers process stats realtime and insert them into sharded
Redis cluster
• Using HINCRBY makes it easy to increment counters
• Updates are performed in bulks (batches)
• Using Redis TTL to automagically delete data after X days
(while the bulk processing has ﬁnished)
• 10s of GB of data, sharded using twemproxy.

The document compares the performance of NFS, GFS2, and OCFS2 filesystems on a high-performance computing cluster with nodes split across two datacenters. Generic load testing showed that NFS performance declined significantly with more than 6 nodes, while GFS2 maintained higher throughput. Further testing of GFS2 and OCFS2 using workload simulations modeling researcher usage found that OCFS2 outperformed GFS2 on small file operations and maintained high performance across nodes, making it the best choice for the shared filesystem needs of the project.

Migrating from InnoDB and HBase to MyRocks at FacebookMariaDB plc

Migrating large databases at Facebook from InnoDB to MyRocks and HBase to MyRocks resulted in significant space savings of 2-4x and improved write performance by up to 10x. Various techniques were used for the migrations such as creating new MyRocks instances without downtime, loading data efficiently, testing on shadow instances, and promoting MyRocks instances as masters. Ongoing work involves optimizations like direct I/O, dictionary compression, parallel compaction, and dynamic configuration changes to further improve performance and efficiency.

Pushing Packets - How do the ML2 Mechanism Drivers Stack UpJames Denton

Architecting a private cloud to meet the use cases of its users can be a daunting task. How do you determine which of the many L2/L3 Neutron plugins and drivers to implement? Does network performance outweigh reliability? Are overlay networks just as performant as VLAN networks? The answers to these questions will drive the appropriate technology choice. In this presentation, we will look at many of the common drivers built around the ML2 framework, including LinuxBridge, OVS, OVS+DPDK, SR-IOV, and more, and will provide performance data to help drive decisions around selecting a technology that's right for the situation. We will discuss our experience with some of these technologies, and the pros and cons of one technology over another in a production environment.

Building large scale transactional data lake using apache hudiBill Liu

Data is a critical infrastructure for building machine learning systems. From ensuring accurate ETAs to predicting optimal traffic routes, providing safe, seamless transportation and delivery experiences on the Uber platform requires reliable, performant large-scale data storage and analysis. In 2016, Uber developed Apache Hudi, an incremental processing framework, to power business critical data pipelines at low latency and high efficiency, and helps distributed organizations build and manage petabyte-scale data lakes. In this talk, I will describe what is APache Hudi and its architectural design, and then deep dive to improving data operations by providing features such as data versioning, time travel. We will also go over how Hudi brings kappa architecture to big data systems and enables efficient incremental processing for near real time use cases. Speaker: Satish Kotha (Uber) Apache Hudi committer and Engineer at Uber. Previously, he worked on building real time distributed storage systems like Twitter MetricsDB and BlobStore. website: https://www.aicamp.ai/event/eventdetails/W2021043010

MyRocks Deep DiveYoshinori Matsunobu

Understanding Storage I/O Under LoadScyllaDB

There’s a popular misconception about I/O that (modern) SSDs are easy to deal with; they work pretty much like RAM but use a “legacy” submit-complete API. And other than keeping in mind a disk’s possible peak performance and maybe maintaining priorities of different IO streams there’s not much to care about. This is not quite the case – SSDs do show non-linear behavior and understanding the disk’s real abilities is crucial when it comes to squeezing as much performance from it as possible. Diskplorer is an open-source disk latency/bandwidth exploring toolset. By using Linux fio under the hood it runs a battery of measurements to discover performance characteristics for a specific hardware configuration, giving you an at-a-glance view of how server storage I/O will behave under load. ScyllaDB CTO Avi Kivity will share an interesting approach to measuring disk behavior under load, give a walkthrough of Diskplorer and explain how it’s used. With the elaborated model of a disk at hand, it becomes possible to build latency-oriented I/O scheduling that cherry-picks requests from the incoming queue keeping the disk load perfectly Balanced. ScyllaDB engineer Pavel Emelyanov will also present the scheduling algorithm developed for the Seastar framework and share results achieved using it.

Kafka 101Clement Demonchy

Advanced Change Data Streaming Patterns in Distributed Systems | Gunnar Morli...HostedbyConfluent

Microservices are one of the big trends in software engineering of the last few years; organising business functionality in several self-contained, loosely coupled services helps teams to work efficiently, make the most suitable technical decisions, and react quickly to new business requirements. In this session we'll discuss and showcase how open-source change data capture (CDC) with Debezium can help developers with typical challenges they often face when working on microservices. Come and join us to learn how to: - Employ the outbox pattern for reliable, eventually consistent data exchange between microservices, without incurring unsafe dual writes or tight coupling - Gradually extract microservices from existing monolithic applications, using CDC and the strangler fig pattern - Coordinate long-running business transactions across multiple services using CDC-based saga orchestration, ensuring such activity gets consistently applied or aborted by all participating services.

Caching solutions with RedisGeorge Platon

Redis is an open source, in-memory data structure store that can be used as a database, cache, or message broker. It supports basic data types like strings, hashes, lists, and sets. Redis features high performance, replication, publishing/subscribing, and Lua scripting. It is widely adopted by companies like GitHub, StackOverflow, and Blizzard for use cases like caching, sessions, queues, and as a real-time database.

Introduction to RedisArnab Mitra

Redis is an open source, in-memory data structure store that can be used as a database, cache, or message broker. It supports data structures like strings, hashes, lists, sets, sorted sets with ranges and pagination. Redis provides high performance due to its in-memory storage and support for different persistence options like snapshots and append-only files. It uses client/server architecture and supports master-slave replication, partitioning, and failover. Redis is useful for caching, queues, and other transient or non-critical data.

Monitoring Apache Kafka with Confluent Control Center confluent

Presentation by Nick Dearden, Direct, Product and Engineering, Confluent It’s 3 am. Do you know how your Kafka cluster is doing? With over 150 metrics to think about, operating a Kafka cluster can be daunting, particularly as a deployment grows. Confluent Control Center is the only complete monitoring and administration product for Apache Kafka and is designed specifically for making the Kafka operators life easier. Join Confluent as we cover how Control Center is used to simplify deployment, operability, and ensure message delivery. Watch the recording: https://www.confluent.io/online-talk/monitoring-and-alerting-apache-kafka-with-confluent-control-center/

RocksDB compactionMIJIN AN

The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.

Introduction to Apache ZooKeeperSaurav Haloi

Redis clusteriammutex

Redis Cluster is an approach to distributing Redis across multiple nodes. Key-value pairs are partitioned across nodes using consistent hashing on the key's hash slot. Nodes specialize as masters or slaves of data partitions for redundancy. Clients can query any node, which will redirect requests as needed. Nodes continuously monitor each other to detect and address failures, maintaining availability as long as each partition has at least one responsive node. The redis-trib tool is used to setup, check, resize, and repair clusters as needed.

Hyperspace for Delta LakeDatabricks

Hyperspace is a recently open-sourced (https://github.com/microsoft/hyperspace) indexing sub-system from Microsoft. The key idea behind Hyperspace is simple: Users specify the indexes they want to build. Hyperspace builds these indexes using Apache Spark, and maintains metadata in its write-ahead log that is stored in the data lake. At runtime, Hyperspace automatically selects the best index to use for a given query without requiring users to rewrite their queries. Since Hyperspace was introduced, one of the most popular asks from the Spark community was indexing support for Delta Lake. In this talk, we present our experiences in designing and implementing Hyperspace support for Delta Lake and how it can be used for accelerating queries over Delta tables. We will cover the necessary foundations behind Delta Lake’s transaction log design and how Hyperspace enables indexing support that seamlessly works with the former’s time travel queries.

HBase Advanced - Lars GeorgeJAX London

Introduction to memcachedJurriaan Persyn

NoSqlGirish Khanzode

hbaseconasia2017: HBase Practice At XiaoMiHBaseCon

Apache Spark & StreamingFernando Rodriguez

대용량 분산 아키텍쳐 설계 #3 대용량 분산 시스템 아키텍쳐Terry Cho

Apache ZooKeeperScott Leberknight

RocksDB detailMIJIN AN

RocksDB is an embedded key-value store written in C++ and optimized for fast storage environments like flash or RAM. It uses a log-structured merge tree to store data by writing new data sequentially to an in-memory log and memtable, periodically flushing the memtable to disk in sorted SSTables. It reads from the memtable and SSTables, and performs background compaction to merge SSTables and remove overwritten data. RocksDB supports two compaction styles - level style, which stores SSTables in multiple levels sorted by age, and universal style, which stores all SSTables in level 0 sorted by time.

Redisimalik8088

Redis is an in-memory key-value database that is used by companies like Instagram and Stack Overflow for caching and storing non-relational data. It supports common data structures like strings, hashes, lists, sets and sorted sets. Redis provides very fast performance of over 100,000 writes and 80,000 reads per second. While Redis does not support complex queries, it is well suited for simple, fast queries and retrievals. The document discusses how Redis could be used to query and retrieve flight booking data by airport in both coarse and fine-grained data modeling approaches.

What every data programmer needs to know about disksiammutex

Disk I/O is much slower than memory access. When reading from a file, the OS first checks the page cache in memory, which has nanosecond access times, before needing to physically access the disk with millisecond seek times. When writing to a file, pages are marked as dirty in the cache before being written to disk asynchronously by flush threads. Fsync forces an immediate write to disk, but caches can lie and data may not be safely written. Virtualized environments add additional layers of caching that can affect performance unpredictably. Hardware techniques like SSDs, RAID controllers, and dedicated flash cards can improve I/O speeds.

SQL-on-Hadoop TutorialDaniel Abadi

This document provides an overview of a SQL-on-Hadoop tutorial. It introduces the presenters and discusses why SQL is important for Hadoop, as MapReduce is not optimal for all use cases. It also notes that while the database community knows how to efficiently process data, SQL-on-Hadoop systems face challenges due to the limitations of running on top of HDFS and Hadoop ecosystems. The tutorial outline covers SQL-on-Hadoop technologies like storage formats, runtime engines, and query optimization.

How to Manage Scale-Out Environments with MariaDB MaxScaleMariaDB plc

MaxScale is a database proxy that provides load balancing, connection pooling, and replication capabilities for MariaDB and MySQL databases. It can be used to scale databases horizontally across multiple servers for increased performance and availability. The document provides an overview of MaxScale concepts and capabilities such as routing, filtering, security features, and how it can be used for operational tasks like query caching, logging, and data streaming. It also includes instructions on setting up MaxScale with a basic example of configuring read/write splitting between a master and slave database servers.

From Mainframe to Microservice: An Introduction to Distributed SystemsTyler Treat

An introductory overview of distributed systems—what they are and why they're difficult to build. We explore fundamental ideas and practical concepts in distributed programming. What is the CAP theorem? What is distributed consensus? What are CRDTs? We also look at options for solving the split-brain problem while considering the trade-off of high availability as well as options for scaling shared data.

Everything you always wanted to know about Redis but were afraid to askCarlos Abalde

Twitter Fatcacheits_skm

Fatcache is an in-memory cache that persists data to solid state drives (SSDs) for faster access than traditional disks. It uses SSDs to provide cheaper and larger storage than memory alone. Fatcache aggregates writes and performs them in batches to SSDs to improve performance and lifespan. It can scale across multiple machines and SSDs for high throughput. Fatcache supports the Memcache protocol and consistent hashing for high availability during failures. However, data is ultimately not durable and could be lost during failures or evictions.

More Related Content

What's hot (20)

Caching solutions with RedisGeorge Platon

Introduction to RedisArnab Mitra

Monitoring Apache Kafka with Confluent Control Center confluent

RocksDB compactionMIJIN AN

Introduction to Apache ZooKeeperSaurav Haloi

Redis clusteriammutex

Hyperspace for Delta LakeDatabricks

HBase Advanced - Lars GeorgeJAX London

Introduction to memcachedJurriaan Persyn

NoSqlGirish Khanzode

hbaseconasia2017: HBase Practice At XiaoMiHBaseCon

Apache Spark & StreamingFernando Rodriguez

대용량 분산 아키텍쳐 설계 #3 대용량 분산 시스템 아키텍쳐Terry Cho

Apache ZooKeeperScott Leberknight

RocksDB detailMIJIN AN

Redisimalik8088

What every data programmer needs to know about disksiammutex

SQL-on-Hadoop TutorialDaniel Abadi

How to Manage Scale-Out Environments with MariaDB MaxScaleMariaDB plc

From Mainframe to Microservice: An Introduction to Distributed SystemsTyler Treat

Caching solutions with RedisGeorge Platon

Introduction to RedisArnab Mitra

Monitoring Apache Kafka with Confluent Control Center confluent

RocksDB compactionMIJIN AN

Introduction to Apache ZooKeeperSaurav Haloi

Redis clusteriammutex

Hyperspace for Delta LakeDatabricks

HBase Advanced - Lars GeorgeJAX London

Introduction to memcachedJurriaan Persyn

NoSqlGirish Khanzode

hbaseconasia2017: HBase Practice At XiaoMiHBaseCon

Apache Spark & StreamingFernando Rodriguez

대용량 분산 아키텍쳐 설계 #3 대용량 분산 시스템 아키텍쳐Terry Cho

Apache ZooKeeperScott Leberknight

RocksDB detailMIJIN AN

Redisimalik8088

What every data programmer needs to know about disksiammutex

SQL-on-Hadoop TutorialDaniel Abadi

How to Manage Scale-Out Environments with MariaDB MaxScaleMariaDB plc

From Mainframe to Microservice: An Introduction to Distributed SystemsTyler Treat

Viewers also liked (20)

Everything you always wanted to know about Redis but were afraid to askCarlos Abalde

Twitter Fatcacheits_skm

Managing Redis with Kubernetes - Kelsey Hightower, GoogleRedis Labs

Redis Use Patterns (DevconTLV June 2014)Itamar Haber

Scaling Redis Cluster Deployments for Genome Analysis (featuring LSU) - Terry...Redis Labs

Neo4jTuan Bq

1 số ứng dụng của Redis, NoSQL tại MXH Tamtay.vnAiTi Education

RedisWei Weng

Redis : Play buzz uses RedisRedis Labs

Chen Salomon introduces himself as a data architect at Playbuzz, a 3-year-old company with 100 employees and 30 in R&D. Playbuzz creates engaging content formats for over 7,000 publishers. To scale efficiently, Playbuzz uses Redis for distributed caching, ETL session management, IP location lookups, and more. Their most recent Redis project converts IP addresses to integers for fast location lookups from cached hashmaps.

Scaling with Riak at ShowyouJohn Muellerleile

Modelling Data as Graphs (Neo4j)Michal Bachman

An intro to Neo4j and some use cases (JFokus 2011)Emil Eifrem

Riak Use Cases : Dissecting The Solutions To Hard ProblemsAndy Gross

This document summarizes a presentation on using Riak to solve hard distributed systems problems. It discusses Riak's key features and tradeoffs in consistency and data model. It then provides examples of how companies like Comcast, Yammer, and Github use Riak for high availability and low latency applications like user metadata storage, notifications, and as a document store. The presentation emphasizes that choosing the right database requires understanding your problem and tradeoffs in consistency, availability and data model. It also highlights how Riak and Erlang help enable high availability, low latency and operational simplicity through features like Bitcask storage, fault tolerance and lightweight threading.

Recommendations with Neo4j (FOSDEM 2015)Michal Bachman

This document discusses recommendations engines that use graph databases like Neo4j. It introduces GraphAware, an open-source recommendation engine plugin for Neo4j. The document outlines the business and technical challenges of building recommendation engines, and how GraphAware addresses these challenges through its flexible, high-performance architecture and APIs. It provides an example of building a simple friend recommendation engine using GraphAware.

Advanced Neo4j Use Cases with the GraphAware FrameworkMichal Bachman

The document discusses GraphAware Framework, which makes it easy to build, test, and deploy custom APIs, transaction-driven behavior, and asynchronous computation functionality for Neo4j. It provides examples like representing time series data, tracking graph changes, assigning UUIDs, and running algorithms. GraphAware Framework is open source and supports building both generic and domain-specific Neo4j extensions.

Redis use casesBottom Wang

Modelling Data in Neo4j (plus a few tips)Michal Bachman

Best Buy Web 2.0Lee Aase

Introduction to Redis Data Structures: Sorted SetsScaleGrid.io

Redis and its many use casesChristian Joudrey

Redis is an open source, advanced key-value store that can be used for caching, sessions, publishing/subscribing, job queues, and more. It supports atomic operations, ordered lists, sets, hashes and sorted sets. Data can be saved to disk for persistence. Redis has client libraries for many languages like Node.js and can be accessed via a command line interface. Popular use cases include caching application data to improve performance, storing sessions to allow user authentication across requests, and processing jobs asynchronously in the background via pub/sub and queues.

Everything you always wanted to know about Redis but were afraid to askCarlos Abalde

Twitter Fatcacheits_skm

Managing Redis with Kubernetes - Kelsey Hightower, GoogleRedis Labs

Redis Use Patterns (DevconTLV June 2014)Itamar Haber

Scaling Redis Cluster Deployments for Genome Analysis (featuring LSU) - Terry...Redis Labs

Neo4jTuan Bq

1 số ứng dụng của Redis, NoSQL tại MXH Tamtay.vnAiTi Education

RedisWei Weng

Redis : Play buzz uses RedisRedis Labs

Scaling with Riak at ShowyouJohn Muellerleile

Modelling Data as Graphs (Neo4j)Michal Bachman

An intro to Neo4j and some use cases (JFokus 2011)Emil Eifrem

Riak Use Cases : Dissecting The Solutions To Hard ProblemsAndy Gross

Recommendations with Neo4j (FOSDEM 2015)Michal Bachman

Advanced Neo4j Use Cases with the GraphAware FrameworkMichal Bachman

Redis use casesBottom Wang

Modelling Data in Neo4j (plus a few tips)Michal Bachman

Best Buy Web 2.0Lee Aase

Introduction to Redis Data Structures: Sorted SetsScaleGrid.io

Redis and its many use casesChristian Joudrey

Similar to Using Redis at Facebook (20)

RedisConf18 - Redis at LINE - 25 Billion Messages Per DayRedis Labs

LINE uses Redis for caching and primary storage of messaging data. It operates over 60 Redis clusters with over 1,000 machines and 10,000 nodes to handle 25 billion messages per day. LINE developed its own Redis client and monitoring system to support client-side sharding without a proxy, automated failure detection, and scalable cluster monitoring. While the official Redis Cluster was tested, it exhibited some issues around memory usage and maximum node size for LINE's large scale needs.

Top ten-listBrian DeShong

This document provides a top ten list of tips for improving PHP and web application performance. They include tweaking realpath cache settings, using offline processing whenever possible, writing efficient SQL queries, not executing queries in loops, caching data, using a content delivery network, and using APC caching with apc.stat set to 0. The tips cover optimizing PHP, database, and infrastructure performance.

Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)VMware Tanzu

Recorded at SpringOne2GX 2013 in Santa Clara, CA Speaker: Adam Shook This session assumes absolutely no knowledge of Apache Hadoop and will provide a complete introduction to all the major aspects of the Hadoop ecosystem of projects and tools. If you are looking to get up to speed on Hadoop, trying to work out what all the Big Data fuss is about, or just interested in brushing up your understanding of MapReduce, then this is the session for you. We will cover all the basics with detailed discussion about HDFS, MapReduce, YARN (MRv2), and a broad overview of the Hadoop ecosystem including Hive, Pig, HBase, ZooKeeper and more. Learn More about Spring XD at: http://projects.spring.io/spring-xd Learn More about Gemfire XD at: http://www.gopivotal.com/big-data/pivotal-hd

RedisHung-yu Lin

Redis is a fast, in-memory key-value database that supports data types like strings, lists, sets and hashes. It features publish-subscribe capabilities, transactions, replication, persistence and partitioning. It is open source and offers clients for many programming languages. Major companies like GitHub, Twitter, Pinterest, and others use Redis for caching, pub/sub messaging and more due to its speed and feature set.

Introduction to Hadoop and Big DataJoe Alex

This document provides an introduction to Hadoop and big data. It discusses the new kinds of large, diverse data being generated and the need for platforms like Hadoop to process and analyze this data. It describes the core components of Hadoop, including HDFS for distributed storage and MapReduce for distributed processing. It also discusses some of the common applications of Hadoop and other projects in the Hadoop ecosystem like Hive, Pig, and HBase that build on the core Hadoop framework.

HBase Low LatencyDataWorks Summit

This document discusses techniques for improving latency in HBase. It analyzes the write and read paths, identifying sources of latency such as networking, HDFS flushes, garbage collection, and machine failures. For writes, it finds that single puts can achieve millisecond latency while streaming puts can hide latency spikes. For reads, it notes cache hits are sub-millisecond while cache misses and seeks add latency. GC pauses of 25-100ms are common, and failures hurt locality and require cache rebuilding. The document outlines ongoing work to reduce GC, use off-heap memory, improve compactions and caching to further optimize for low latency.

Google file systemDhan V Sagar

The document summarizes Google File System (GFS), which was designed to provide reliable, scalable storage for Google's large data processing needs. GFS uses a master server to manage metadata and chunk servers to store file data in large chunks (64MB). It replicates chunks across multiple servers for reliability. The architecture supports high throughput by minimizing interaction between clients and the master, and allowing clients to read from the closest replica of a chunk.

BSides Algiers - Layer7 DoS Attacks - Oussama ElhamerShellmates

This document discusses different types of layer 7 denial of service (DOS) attacks, including HTTP GET, HTTP POST, Slowloris, and R-U-Dead-Yet attacks. It also covers layer 4 DDOS attacks and link-local DOS attacks using IPv6 router advertisements. Layer 7 DOS attacks can be more difficult to detect than layer 4 attacks because they operate at the application layer and can be routed through proxies with low bandwidth. Examples provided show how incomplete requests can tie up server resources and bring down sites with a single attacker.

Hadoop Meetup Jan 2019 - HDFS Scalability and Consistent Reads from Standby NodeErik Krogen

Running php on nginxHarald Zeitlhofer

The document discusses configuring Nginx and PHP-FPM for high performance websites. Some key points: - Nginx is a lightweight and fast HTTP server that is well-suited for high traffic loads. It can be used as a web server, reverse proxy, load balancer, and more. - PHP-FPM (PHP FastCGI Process Manager) runs PHP processes as a pool that is separate from the web server for better isolation and performance. Nginx communicates with PHP-FPM via FastCGI. - Benchmark results show Nginx performing better than Apache, especially under high concurrency loads. Caching with Nginx and Memcached can further improve

HIgh Performance Redis- Tague Griffith, GoProRedis Labs

High Performance Redis looks at a wide range of techniques - from programming to system tuning - to deploy and maintain an extremely high performing Redis cluster. From the operational perspective, the talk lays out multiple techniques for clustering (sharding) Redis systems and examines how the different approaches impact performance time. The talk further examines system settings (Linux network parameters, Redis system) and how they impact performance (both good and bad). Finally, for the developer, we look at how different ways of structuring data actually demonstrate different performance characteristics

HBase Low Latency, StrataNYC 2014Nick Dimiduk

Bigdata workshop february 2015 clairvoyantllc

This document provides an overview of big data and Hadoop. It discusses what big data is, why it has become important recently, and common use cases. It then describes how Hadoop addresses challenges of processing large datasets by distributing data and computation across clusters. The core Hadoop components of HDFS for storage and MapReduce for processing are explained. Example MapReduce jobs like wordcount are shown. Finally, higher-level tools like Hive and Pig that provide SQL-like interfaces are introduced.

HadoopGirish Khanzode

Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It implements Google's MapReduce programming model and the Hadoop Distributed File System (HDFS) for reliable data storage. Key components include a JobTracker that coordinates jobs, TaskTrackers that run tasks on worker nodes, and a NameNode that manages the HDFS namespace and DataNodes that store application data. The framework provides fault tolerance, parallelization, and scalability.

MongoDB World 2018: Active-Active Application Architectures: Become a MongoDB...MongoDB

MongoDB can be configured to meet the requirements of active-active applications across multiple data centers. There are three main deployment patterns: 1) active-passive with one data center as primary, 2) partitioned databases with each data center owning a partition, and 3) multi-master with each data center acting as a master. The document discusses how to tune MongoDB for performance, consistency, availability, and durability using features like sharding, read preference, write concern, and causal consistency.

Improving HDFS Availability with Hadoop RPC Quality of ServiceMing Ma

Heavy users monopolizing cluster resources is a frequent cause of slowdown for others. With only one namenode and thousands of datanodes, any poorly written application is a potential distributed denial-of-service attack on namenode. In this talk, you will learn how to prevent slowdown from heavy users and poorly-written applications by enabling IPC Quality of Service (QoS), a new feature in Hadoop 2.6+. On Twitter’s and eBay’s production clusters, we’ve seen response times of 500 milliseconds with QoS off drop to 10 milliseconds with QoS on during heavy usage. We’ll cover how IPC QoS works and share our experience on how to tune performance.

Redis acc 2015_engDaeMyung Kang

This document summarizes Redis versions from 2.8 to 3.2 and discusses new features and improvements. Redis 2.8 introduced the SCAN command to iteratively scan keys and partial sync replication to avoid full resynchronization if connections are briefly lost. Redis 3.0 added Redis Cluster for automatic sharding and diskless replication for faster replication without disk I/O. Redis 3.2 optimized string header sizes in SDS to reduce memory usage and added new GEO commands.

Introduction to HDFS and MapReduceDerek Chen

Aziksa hadoop architecture santosh jhaData Con LA

Hadoop is an open-source framework for distributed storage and processing of large datasets across clusters of commodity hardware. It has several core components including HDFS for distributed file storage and MapReduce for distributed processing. HDFS stores data across clusters of machines with replication for fault tolerance. MapReduce allows parallel processing of large datasets in a distributed manner. Hadoop was designed with goals of using commodity hardware, easy recovery from failures, large distributed file systems, and fast processing of large datasets.

Redis - The Universal NoSQL ToolEberhard Wolff

Redis is a fast, in-memory key-value store that can be used as a cache, message broker, and centralized locking system. It stores data in memory for high performance, but can also asynchronously write data to disk for persistence. Redis supports data types like strings, lists, sets and sorted sets. It enables high availability through master-slave replication and horizontal scaling through sharding of data across multiple nodes.

RedisConf18 - Redis at LINE - 25 Billion Messages Per DayRedis Labs

Top ten-listBrian DeShong

Hadoop - Just the Basics for Big Data Rookies (SpringOne2GX 2013)VMware Tanzu

RedisHung-yu Lin

Introduction to Hadoop and Big DataJoe Alex

HBase Low LatencyDataWorks Summit

Google file systemDhan V Sagar

BSides Algiers - Layer7 DoS Attacks - Oussama ElhamerShellmates

Hadoop Meetup Jan 2019 - HDFS Scalability and Consistent Reads from Standby NodeErik Krogen

Running php on nginxHarald Zeitlhofer

HIgh Performance Redis- Tague Griffith, GoProRedis Labs

HBase Low Latency, StrataNYC 2014Nick Dimiduk

Bigdata workshop february 2015 clairvoyantllc

HadoopGirish Khanzode

MongoDB World 2018: Active-Active Application Architectures: Become a MongoDB...MongoDB

Improving HDFS Availability with Hadoop RPC Quality of ServiceMing Ma

Redis acc 2015_engDaeMyung Kang

Introduction to HDFS and MapReduceDerek Chen

Aziksa hadoop architecture santosh jhaData Con LA

Redis - The Universal NoSQL ToolEberhard Wolff

More from Redis Labs (20)

Redis Day Bangalore 2020 - Session state caching with redisRedis Labs

This document discusses using Redis caching to improve performance for the DBS Paylah mobile wallet application. Paylah aims to significantly increase its user base which will increase load on its backend systems. Caching application data and session state in Redis can reduce latency, improve responsiveness for users, and reduce costs by lowering load on legacy backend databases and mainframes. The document outlines some key Paylah use cases where caching transaction histories and account details in Redis would accelerate retrieval and improve the mobile experience by avoiding the need to access slower backend systems on each request.

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020Redis Labs

The document discusses rate limiting and metering using Redis. It begins by introducing rate limiting and metering and why Redis is well-suited for these tasks. It then covers different Redis data structures that can be used, such as lists, hashes, sorted sets and strings. Common Redis commands for counting, setting keys and checking time to live are also presented. Different rate limiting design patterns and anti-patterns are described, including fixed window, sliding window and token bucket approaches. Finally, resources for further information are provided.

The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...Redis Labs

The document summarizes a presentation about using Protocol Buffers and Redis together. It discusses how Protocol Buffers provide strict data types, versioning, and serialization/deserialization benefits. It then outlines Redis key patterns using namespaces, versions, data categories and identifiers. Examples are provided to show how Protocol Buffers messages can be stored in Redis using these key patterns, including storing connections data in sets, sorted sets and individual messages. Benefits discussed include structure, readability, testing and abstraction.

SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020Redis Labs

The document discusses common use cases for combining SQL, Redis, and Kubernetes including caching, session management, rate limiting, and data ingestion. It outlines how Kubernetes can be used for scaling microservices while Redis is used for data service scaling. The presentation proposes combining Redis, SQL Server, and Kubernetes with a proxy service, and describes using Redis for caching, session storage, and rate limiting of SQL data. It also discusses running Redis and front-end apps on Kubernetes and deploying SQL as a Kubernetes service through a proxy.

Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...Redis Labs

This document discusses using Rust and Redis to build cloud native platforms. It first provides context about devops and the need to do more with less. It then discusses how platforms are becoming more distributed and Kubernetes upends distribution paradigms. The document dives into how Rust addresses issues like concurrency and systems programming. It also discusses how Redis can be used for caching, queues, streams and more. Finally, it mentions that Rust and Redis will be demonstrated.

Redis for Data Science and Engineering by Dmitry Polyakovsky of OracleRedis Labs

This document contains a presentation about using Redis for data science and engineering. It introduces the presenter and provides an agenda that covers using Redis for data science and data engineering. The presentation notes that Redis can be used as both a data store and job queue, has flexible data structures and is fast, though it uses RAM and cannot query by value. It also lists Python Pandas and includes a demo and links for further information.

Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020Redis Labs

Jamie Scott from RedisLabs presented on practical use cases for access control lists (ACLs) in Redis 6. The presentation covered new security features in Redis 6 including encryption in transit, key space and command restrictions, and multiple access control list users. It demonstrated how ACLs allow users to define access based on key labels and restrictions. ACLs can facilitate discretionary and mandatory access controls. The presentation showed examples of using ACLs to restrict user access by key labels and commands to enhance operational security.

Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020Redis Labs

This document summarizes a presentation about Redis version 6 and beyond. Some key points include: - Redis version 6 includes new features like ACL for security, client-side caching, diskless replication, and multi-threaded I/O. - Redis is positioned as both a cache and a database due to its speed, data structures, and ability to handle complex data models through modules. - Redis Enterprise provides additional capabilities like durability, high availability, geo-distribution, security and multi-tenancy. - Modern data models in Redis modules include Streams, RediSearch, RedisGraph, RedisTimeSeries, RedisAI, RedisJSON and RedisBloom. - RedisInsight is

Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...Redis Labs

The document discusses how Sinclair Broadcast Group leverages Redis for system monitoring of its content delivery network. It operates 193 news stations with 10,000 active pages daily and millions in archive. New stories are posted every 15 seconds and must be visible across its 1,000+ targets within 1 minute. Redis is used to track performance across the multi-level CDN and ensure service level agreements are met with real-time resolution and alerting. It provides a black box view of the audience experience and can scale monitoring to all relevant pages within 30 seconds. Redis acts as a distributed data store to parallelize the monitoring task across the large scale of the network.

JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...Redis Labs

The document summarizes a presentation about when to use the RedisJSON data type. It discusses how Coupang uses Redis extensively for their ad platform. It then compares the performance and memory usage of storing JSON data as strings, hashes, or using the RedisJSON data type. Benchmark results show RedisJSON can provide better performance for retrieving and updating JSON fields compared to strings and hashes, though it uses more memory. The document recommends using RedisJSON for smaller JSON payloads after benchmarking and memory monitoring.

Highly Available Persistent Session Management Service by Mohamed Elmergawi o...Redis Labs

The document discusses the challenges of building a highly available persistent session management service. It describes Zulily's legacy architecture which lacked high availability and required manual intervention. A new architecture is proposed using Redis for persistent storage, Dynomite for real-time replication across data centers, and a connection pooling proxy to improve efficiency and distribute load. The architecture provides high availability through replication, reduces overhead through connection pooling, and handles failures through consistent hashing and health checks. It was tested through simulations and showed a failure rate of only 0.42% during outages.

Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...Redis Labs

The document describes the process that a Redis command follows from the client side to the server side. On the client side, the command is sent over the network to the Redis server. On the server side, the command is read from the kernel buffers, validated, executed by calling the relevant command handler, and the response is written back to the client over the network. The core functions involved on the server side are ReadQueryFromClient(), ProcessInputBuffer(), ProcessCommand(), Call(), and handleClientsWithPendingWrites(). Redis 6.0 introduced I/O threads to handle reads and writes in parallel for improved performance while still maintaining Redis' single-threaded processing model.

Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...Redis Labs

This document discusses MDmetrix, a healthcare data intelligence company that uses RedisGraph to provide flexible analysis of hospital data. RedisGraph is a graph database that represents data as nodes and relationships and uses an adjacency matrix and linear algebra to query the graph. MDmetrix models its healthcare data as a property graph in RedisGraph to allow for complex queries across different data dimensions like patients, facilities, procedures and drugs. RedisGraph allows MDmetrix to query the data more easily than traditional OLAP cubes or relational databases due to the semi-structured and flexible nature of the graph model.

RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020Redis Labs

RediSearch 1.6 includes a new low-level API that allows other Redis modules to embed RediSearch indexing capabilities. It also introduces index aliasing and several performance improvements such as forked thread garbage collection. Based on benchmarks, RediSearch 1.6 shows 48-73% better performance than version 1.4, particularly during high update rates where it maintains more stable read latencies.

RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020Redis Labs

RedisGraph 2.0 provides significant improvements including: - Full text search support through embedded RediSearch 1.6 enabling graph-aided search. - Support for returning full graph responses to enable better visualization. - Broad support for Cypher including triadic closure and new graph-aided search capabilities. - Performance improvements of up to 3.7x faster operations per second and 3.9x faster query times compared to RedisGraph v1.2. - Support for benchmarking including the LDBC benchmark.

RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020Redis Labs

RedisTimeSeries is a time-series database that provides compression to reduce memory usage by up to 98% and improve performance. The RedisTimeSeries 1.2 release includes compression algorithms based on a Facebook paper that provide stable ingestion times independent of the number of data points. It also includes a reviewed API with performance improvements and clearer functionality. Performance testing showed ingestion throughput improved by 2-3% and query performance increased from 15-70% with the new release compared to the previous version.

RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020Redis Labs

This document summarizes RedisAI 0.9 and its capabilities for model deployment and benchmarking. It introduces RedisAI's new tensor data type and ability to deploy models to CPU and GPU. It then discusses AIBench, a tool developed to benchmark AI serving solutions like RedisAI, TensorFlow Serving, and REST APIs. The benchmarks show RedisAI providing 5.5x and 2.5x more inferences than REST APIs and TensorFlow Serving respectively, due to its data locality. The document concludes by mentioning RedisAI's integration with MLFlow for model deployment with a single command.

Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...Redis Labs

The document discusses how Freshworks uses Redis Labs to rate limit 30 million API requests per day through their API gateway called Fluffy. Fluffy stores rate limit policies and maintains counters to track requests. Redis Labs allows Fluffy to easily scale to handle the high volume of requests by providing a fast, in-memory data store for managing rate limiting counters. The system was able to successfully rate limit 30 million requests per day with Redis Labs.

Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...Redis Labs

Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...Redis Labs

Redis was used by Myntra to solve several complex scaling problems. It was used to build a scalable user segment service to support high read throughput of up to 5 million requests per minute with low latency. Redis allowed the service to scale beyond a single instance and included features like automatic backups and memory management. Redis also helped build a scalable mobile verification platform to reliably handle 100,000 requests per minute and scale to support higher future volumes. It was used as both a transient store and persistent backend. Finally, Redis locks helped build a scalable A/B testing platform by allowing experiments to be created and updated in an orderly concurrent fashion.