RedisConf17 - Redis Cluster at flickr and tripod

Jun 2, 2017Download as PPTX, PDF1 like1,069 views

Flickr uses Redis extensively across many services for caching, queueing, and more. Some key uses of Redis include caching for the camera roll and activity feed, which each have multiple masters and slaves distributed across data centers. Notifications are also queued in Redis. In total, Flickr operates over 100 physical hosts running 500+ Redis instances to support over 600,000 operations per second across services like the camera roll, activity feed, notifications, and more.

@ flickr & tripod
Presented by Sean Perkins
Sr. Production Engineer

flickr & tripod
100+ physical hosts
500+ instances
600k Ops/s

Cameraroll
24 hosts (12 on each coast)
4 masters per host
cache
twemproxy client
ketama hash ring

Activityfeed
8 hosts
8 masters per host
cache
twemproxy client
ketama hash ring

Notifications
3 Hosts
1 Master
2 Slaves
Clients on both coasts
Redis only in one

And many many more…
45 redis-general hosts w/ ~200 instances
about 50% have HA

Redis Cluster
Manual builds at first
created package to automate setup
tested failed nodes
benchmarking

redis-cluster101
redis-cluster102
redis-cluster103
196GB RAM each
8 master instances x 3 hosts = 24
16 slave instances x 3 hosts = 48
tripod

Console (tripod on boarding site)
Java Spring Client
Cache
Smart Client

redis-cluster101
redis-cluster102
redis-cluster103
64GB RAM each
4 master instances x 3 hosts = 12
8 slave instances x 3 hosts = 24
flickr

Activitybot
Kue Library and Nodejs
Queue
Smart Client

This document summarizes RedisConf 2017, covering several topics: 1. Running Redis on Flash in a DBaaS model for improved performance and cost savings compared to other NoSQL databases. 2. Redis modules gaining momentum with over 50 created so far, and the importance of multi-threading for high performance. Useful modules highlighted include RediSearch, ReJSON, Redis-ML, and Redis-Graph. 3. Using Redis for IoT applications, with challenges around small edge devices and clusters, high throughput from thousands of devices, and varied functionality needs addressed through Redis modules.

RedisConf17 - Lyft - Geospatial at Scale - Daniel HochmanRedis Labs

This document discusses Lyft's migration from a monolithic architecture to using Redis to power a geospatial indexing system at scale. It describes Lyft's original architecture, issues they faced, and how they iterated on their data model and use of Redis over time. It also discusses how Lyft uses Redis across their platform, including operations, monitoring, capacity planning, and their work contributing to open source projects like Envoy.

RedisConf17- Using Redis at scale @ TwitterRedis Labs

The document discusses Nighthawk, Twitter's distributed caching system which uses Redis. It provides caching services at a massive scale of over 10 million queries per second and 10 terabytes of data across 3000 Redis nodes. The key aspects of Nighthawk's architecture that allow it to scale are its use of a client-oblivious proxy layer and cluster manager that can independently scale and rebalance partitions across Redis nodes. It also employs replication between data centers to provide high availability even in the event of node failures. Some challenges discussed are handling "hot keys" that get an unusually high volume of requests and more efficiently warming up replicas when nodes fail.

RedisConf17 - Redis in High Traffic Adtech StackRedis Labs

Rahul Babbar presented on Times Internet's use of Redis in their high-traffic adtech stack. Some key points: - Times Internet is India's largest digital company with 240+ million unique visitors per month. - Their adtech stack powers ads for 150+ publishers and handles 9 billion ad impressions per month. - Redis is used for caching, user profiles, analytics and more due to its low latency and ability to handle high volumes. - Their Redis clusters consist of over 160 nodes across 4 clusters, serving over 2 million queries per second with 99% under 2ms response times.

What's new with enterprise Redis - Leena Joshi, Redis LabsRedis Labs

Redis Labs manages over 160k+ HA databases, 10k clustered databases, without data loss in spite of one node failure a day and one data center outage per month. Using Enterprise Redis(RLEC), Redis Labs delivers seamless zero downtime scaling, true high availability with persistence, cross-rack/zone/ datacenter replication and instant automatic failover. Learn how. Join this session for a deep dive into how enterprise Redis makes for no-hassle Redis deployments and the roadmap for new Redis capabilities. Discover new cost savings with Redis on Flash for cost-effective high performance operations and analytics

RedisConf17 - Redis Development, An Update - @antirezRedis Labs

Redis 4.0 will include several major new features and improvements including PSYNC2 for better replication, UNLINK for non-blocking DEL operations, and modules to allow building additional functionality on Redis. Streams will be a new data type in Redis 4.2, providing an abstract log file structure for ordered, timestamped entries that allows for efficient random access, deletion of old data, and multiple consumers similar to Kafka. Streams are designed to address use cases like time series data and message queues more efficiently than existing Redis data types.

RedisConf17 - Searching Billions of Documents with RedisRedis Labs

Redis Labs is a company that provides Redis Enterprise, a fully managed Redis database. Redis is an in-memory database but lacks search capabilities. Redis Labs developed RediSearch, a Redis module that allows for fast searching of billions of documents stored in Redis. RediSearch uses inverted indexes and custom data structures to enable full-text search, numeric filtering, and other search features in Redis in a memory-efficient way. It can also scale to handle large indexes by partitioning documents across multiple Redis servers and coordinating queries.

RedisConf17 - Rax, Listpack and Safe ContextsRedis Labs

This document summarizes recent improvements to Redis data structures including Rax, Listpack, and safe threading contexts. Rax is a new radix tree implementation that uses less memory and is faster than the previous dictionary implementation for most use cases. Listpack is a new more memory efficient encoding for lists compared to ziplist. Safe threading contexts allow blocking commands in modules without blocking the main Redis event loop. Contexts provide thread-safe access to the Redis API and ability to reply to clients from threads. Future improvements may include key locking and read-write locks to further improve threading in Redis modules.

Counting image views using redis clusterRedis Labs

Streaming Logs and Processing View Counts using Redis Cluster Seandon Mooy (Imgur) When you browse through Imgur, you notice that each user's post includes the number of views for that particular post. Imgur processes over 3 billion views per month and powers our view count feature using Redis. In this talk, we cover our current architecture for streaming logs and processing view counts using Redis Cluster, as well as some of the alternatives we explored and why we chose Redis.

Troubleshooting redisDaeMyung Kang

This document discusses troubleshooting Redis. Some key points: - Redis is single-threaded, so commands like KEYS, FlushAll, and deleting large collections can be slow. It's better to use SCAN instead of KEYS. - Creating Redis database snapshots (RDB files) and rewriting the append-only file (AOF) can cause high disk I/O and CPU usage. It's best to disable automatic rewrites. - Monitoring memory usage and fragmentation is important to avoid performance issues. The maxmemory setting also needs monitoring to prevent out-of-memory errors. - Network and replication failures need solutions like DNS failover or using Zookeeper for coordination to maintain high availability of Redis

RedisConf17 - Redis GraphRedis Labs

This document discusses Redis Graph, an open source graph database module for Redis. It provides concise summaries of Redis Graph's key features: - Redis Graph allows storing and querying graph-structured data using a graph data model with nodes, edges, and properties. It supports multi-hop queries across relationships. - Redis Labs provides commercial support for Redis Graph through products like Redis Enterprise Cloud and on-premise offerings. - The document demonstrates basic graph queries in Cypher and shows the query execution plan, highlighting Redis Graph's ability to efficiently query large graph datasets.

RedisConf17 - Turbo-charge your apps with Amazon Elasticache for RedisRedis Labs

This document provides an overview and summary of Amazon ElastiCache for Redis. It discusses the key features of ElastiCache including easy deployment and monitoring, enhanced Redis engine capabilities, high availability, cost effectiveness, and integration with other AWS services. It also covers usage patterns such as database caching, streaming data processing, and building real-time apps. Finally, it discusses best practices for building resilient architectures on ElastiCache including reference architectures, failure scenarios, and open source contributions from AWS.

RedisConf17 - Home Depot - Turbo charging existing applications with RedisRedis Labs

The Home Depot is transforming its architecture to use microservices and polyglot persistence to handle increasing online order volumes of 250,000 lines per hour. Redis is being used to turbo charge existing monolithic applications by offloading pieces to new processes using patterns like caching, concurrency management, and powering algorithms. This improves performance by reducing database degradation and wait times by over 95%. Next steps include setting up Redis clusters on-premises and off-premises to further reduce database CPU usage and onboard more patterns.

Dynomite: A Highly Available, Distributed and Scalable Dynamo Layer--Ioannis ...Redis Labs

Dynomite is a thin, distributed dynamo layer for different storage engines and protocols. Currently at Netflix, we are focusing on using Redis as the storage engine. Dynomite supports multi-datacenter replication and is designed for high availability. In the age of high scalability and big data, Dynomite’s design goal is to turn single-server datastore solutions into peer-to-peer, linearly scalable, clustered systems while still preserving the native client/server protocols of the datastores, e.g., Redis protocol. In this talk, we are going to present Dynomite recent features, and the Dyno client. Both projects are open source and available to the community.

MySQL Head to Head PerformanceKyle Bader

This document outlines an agenda for a conference on MySQL and Ceph storage solutions. The agenda includes sessions on MySQL performance on Ceph versus AWS, a head-to-head performance lab comparing the two platforms, and architectural considerations for optimizing MySQL on Ceph. Specific topics covered are MySQL and Ceph capabilities like live migration and snapshots, ensuring a consistent developer experience between private Ceph and public cloud, results from sysbench tests showing Ceph can match or exceed AWS performance on price per IOPS, and how Ceph node configuration like CPU cores and flash storage affect MySQL workload performance.

Which Hypervisor is Best?Kyle Bader

This document summarizes a presentation about tuning MySQL performance on Ceph block storage. The presentation covers Ceph architecture, tuning Ceph block devices, and tuning QEMU block virtualization. It then shows benchmarks comparing different configurations for reads, writes, and a 70/30 read/write mix using Sysbench OLTP workloads. Configurations tested include QEMU backends, caching modes, I/O threading, virtio types and queues, and containers versus metal. The goal is to understand how to optimize MySQL on Ceph block devices.

Day 2 General Session Presentations RedisConfRedis Labs

The document discusses new memory technologies like persistent memory and their implications. It provides latency and bandwidth numbers for different memory types and notes that heterogeneous memory systems using tiers of DRAM and NVM provide opportunities for better performance and cost. Examples are given of key-value stores and databases leveraging NVM to achieve high performance while reducing costs. The talk also discusses how new distributed data structures like CRDTs could be used across servers with shared memory.

Handling Redis failover with ZooKeeperryanlecompte

This document discusses using ZooKeeper to automatically handle Redis failover. ZooKeeper is an open-source tool that provides primitives for building distributed applications and handles tasks like leader election and quorum management. The presenter describes how his redis_failover Ruby gem uses ZooKeeper to monitor Redis servers, detect failures, and automatically inform clients so they reconnect to the new master, preventing downtime during a failover. Several companies already use this approach with redis_failover to make their Redis infrastructure more robust and fault-tolerant.

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...ScyllaDB

Scylla is an open source reimplementation of Cassandra which performs up to 10X with drop in-replacement compatibility. At ScyllaDB, performance matters but even more importantly, stable performance under any circumstances. A key factor for our consistent performance is our reliance on userspace schedulers. Scheduling in userspace allows the application, the database in our case to have better control on the different priorities each task has and to provide an SLA to selected operations. Scylla used to have an I/O scheduler and recently won a CPU scheduler. At ScyllaDB, we make architectural decisions that provide not only low latencies but consistently low latencies at higher percentiles. This begins with our choice of language and key architectural decisions such as not using the Linux page-cache, and is fulfilled by autonomous database control, a set of algorithms, which guarantees that the system will adapt to changes in the workload. In the last year, we have made changes to Scylla that provide latencies that are consistent in every percentile. In this talk, Dor Laor will recap those changes and discuss what ScyllaDB is doing in the future.

RedisConf17 - Building Large High Performance Redis Databases with Redis Ente...Redis Labs

This document discusses building large databases with Redis Enterprise (Redise) using flash memory. It introduces Redis Labs and their Redise product, which uses a clustered architecture to scale Redis deployments. Redise allows scaling data beyond RAM by extending into flash memory at a lower cost than using only RAM. Performance tests show Redise running on Intel Optane SSDs can achieve up to 9x higher throughput than traditional SSDs for large datasets. The document advocates Redise Flash as a cost-effective way to handle massive datasets with near-RAM latency.

Ceph Deployment at Target: Customer SpotlightColleen Corrice

Redis in a Multi Tenant Environment–High Availability, Monitoring & Much More! Redis Labs

This document discusses best practices for running Redis in a multi-tenant environment. It covers architectural considerations like high availability, security and isolation techniques using ACLs and SSL, and the importance of monitoring and understanding your environment. The key opportunities are that failover is easy to implement, changes can be introduced smoothly, and the architecture is reusable. Challenges include managing global sentinels and Redis drivers. Case studies demonstrate issues like customer spikes causing problems and the importance of monitoring.

Persistent Storage for Containerized ApplicationsColleen Corrice

Redis for horizontally scaled data processing at jFrog bintrayRedis Labs

Building Scalable, Real Time Applications for Financial Services with DataStaxDataStax

The document discusses building scalable real-time applications for financial services using Apache Cassandra. It provides an overview of Cassandra's key capabilities including linear scalability, high availability, tunable consistency and multi-datacenter support. It also discusses example architectures for resilient application tiers using Cassandra to store session data across multiple data centers.

HIgh Performance Redis- Tague Griffith, GoProRedis Labs

High Performance Redis looks at a wide range of techniques - from programming to system tuning - to deploy and maintain an extremely high performing Redis cluster. From the operational perspective, the talk lays out multiple techniques for clustering (sharding) Redis systems and examines how the different approaches impact performance time. The talk further examines system settings (Linux network parameters, Redis system) and how they impact performance (both good and bad). Finally, for the developer, we look at how different ways of structuring data actually demonstrate different performance characteristics

Red Hat Storage RoadmapColleen Corrice

Red Hat's storage roadmap aims to offer a unified, open software-defined storage portfolio for next generation workloads. Their portfolio includes Ceph and Gluster storage, which provide data services like object storage, block storage, and file sharing on standard hardware. Red Hat is working to improve management, performance, security, and integration of these products. They are also exploring new workloads like running MySQL on Ceph block storage and using Gluster Storage for hyperconverged Red Hat Enterprise Virtualization environments.

Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Odinot Stanislas

Après la petite intro sur le stockage distribué et la description de Ceph, Jian Zhang réalise dans cette présentation quelques benchmarks intéressants : tests séquentiels, tests random et surtout comparaison des résultats avant et après optimisations. Les paramètres de configuration touchés et optimisations (Large page numbers, Omap data sur un disque séparé, ...) apportent au minimum 2x de perf en plus.

Docker in the WildChristian Mague

This document discusses Docker and its use in development, continuous delivery pipelines, and production environments. It outlines some benefits of Docker like fast startup times and lightweight containers, but also challenges like requiring Linux and only running one process per container. The document recommends using tools from HashiCorp to build Docker containers for development and production that are modular and support services like Consul, databases, and logging. It stresses the importance of monitoring, logging, and immutable infrastructure for Docker in production.

Docker in the WildChris Mague

More Related Content

What's hot (20)

Counting image views using redis clusterRedis Labs

Troubleshooting redisDaeMyung Kang

RedisConf17 - Redis GraphRedis Labs

RedisConf17 - Turbo-charge your apps with Amazon Elasticache for RedisRedis Labs

RedisConf17 - Home Depot - Turbo charging existing applications with RedisRedis Labs

Dynomite: A Highly Available, Distributed and Scalable Dynamo Layer--Ioannis ...Redis Labs

MySQL Head to Head PerformanceKyle Bader

Which Hypervisor is Best?Kyle Bader

Day 2 General Session Presentations RedisConfRedis Labs

Handling Redis failover with ZooKeeperryanlecompte

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...ScyllaDB

RedisConf17 - Building Large High Performance Redis Databases with Redis Ente...Redis Labs

Ceph Deployment at Target: Customer SpotlightColleen Corrice

Redis in a Multi Tenant Environment–High Availability, Monitoring & Much More! Redis Labs

Persistent Storage for Containerized ApplicationsColleen Corrice

Redis for horizontally scaled data processing at jFrog bintrayRedis Labs

Building Scalable, Real Time Applications for Financial Services with DataStaxDataStax

HIgh Performance Redis- Tague Griffith, GoProRedis Labs

Red Hat Storage RoadmapColleen Corrice

Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Odinot Stanislas

Counting image views using redis clusterRedis Labs

Troubleshooting redisDaeMyung Kang

RedisConf17 - Redis GraphRedis Labs

RedisConf17 - Turbo-charge your apps with Amazon Elasticache for RedisRedis Labs

RedisConf17 - Home Depot - Turbo charging existing applications with RedisRedis Labs

Dynomite: A Highly Available, Distributed and Scalable Dynamo Layer--Ioannis ...Redis Labs

MySQL Head to Head PerformanceKyle Bader

Which Hypervisor is Best?Kyle Bader

Day 2 General Session Presentations RedisConfRedis Labs

Handling Redis failover with ZooKeeperryanlecompte

How we got to 1 millisecond latency in 99% under repair, compaction, and flus...ScyllaDB

RedisConf17 - Building Large High Performance Redis Databases with Redis Ente...Redis Labs

Ceph Deployment at Target: Customer SpotlightColleen Corrice

Redis in a Multi Tenant Environment–High Availability, Monitoring & Much More! Redis Labs

Persistent Storage for Containerized ApplicationsColleen Corrice

Redis for horizontally scaled data processing at jFrog bintrayRedis Labs

Building Scalable, Real Time Applications for Financial Services with DataStaxDataStax

HIgh Performance Redis- Tague Griffith, GoProRedis Labs

Red Hat Storage RoadmapColleen Corrice

Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Odinot Stanislas

Similar to RedisConf17 - Redis Cluster at flickr and tripod (20)

Docker in the WildChristian Mague

Docker in the WildChris Mague

Redis Meetup TLV - K8s Session 28/10/2018Danni Moiseyev

- The document discusses deploying Redis Enterprise on Kubernetes using a custom RedisCluster operator. - The operator allows full control over the Redis Enterprise cluster lifecycle and integrates it with Kubernetes. - Key aspects covered include building the deployment with resources like StatefulSets, services, secrets and persistent volumes as well as ensuring high availability, auto-scaling and failover.

IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...In-Memory Computing Summit

Have you heard that all in-memory databases are equally fast but unreliable, inconsistent and expensive? This session highlights in-memory technology that busts all those myths. Redis, the fastest database on the planet, is not a simply in-memory key-value data-store; but rather a rich in-memory data-structure engine that serves the world’s most popular apps. Redis Labs’ unique clustering technology enables Redis to be highly reliable, keeping every data byte intact despite hundreds of cloud instance failures and dozens of complete data-center outages. It delivers full CP system characteristics at high performance. And with the latest Redis on Flash technology, Redis Labs achieves close to in-memory performance at 70% lower operational costs. Learn about the best uses of in-memory computing to accelerate everyday applications such as high volume transactions, real time analytics, IoT data ingestion and more.

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화OpenStack Korea Community

Ceph is an open-source distributed storage system that provides object, block, and file storage. The document discusses optimizing Ceph for an all-flash configuration and analyzing performance issues when using Ceph on all-flash storage. It describes SK Telecom's testing of Ceph performance on VMs using all-flash SSDs and compares the results to a community Ceph version. SK Telecom also proposes their all-flash Ceph solution with custom hardware configurations and monitoring software.

Orchestrating Redis & K8s OperatorsDoiT International

The document discusses an agenda for a meetup about Redis Labs and Kubernetes operators. Key points: - An introduction to Redis Enterprise architecture and Redis Labs products. - A discussion of "double orchestration" using Kubernetes and PKS to manage Redis clusters for performance and resource management. - An overview of Redis Labs' Kubernetes solution using StatefulSets, services, and a custom controller. - An introduction to operators, how they provide lifecycle management and simplify deployments compared to static YAML files or Helm. - Details on Redis Labs' operator development process and challenges in building idempotent APIs and handling state changes and validation in the reconciliation loop.

Docker vs kvmWilson Cunalata

This document compares the performance of KVM and Docker instances in OpenStack. It finds that for boot and delete operations, KVM is around 20% faster than Docker at high concurrency levels. For CPU and memory tests at low concurrency, KVM is 2-10% slower than bare metal, while Docker is 2-5% slower. For sequential and random read/write file I/O tests, KVM is similar or slightly slower than bare metal, while Docker is slightly slower than KVM. Overall performance decreases for both KVM and Docker as concurrency increases.

SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE

This document summarizes a presentation about SUSE Linux Enterprise High Availability Cluster Multi-Device. It discusses the main features of SUSE HA including policy driven clusters, cluster aware filesystems, and continuous data replication. It then describes the HA storage stack architecture and various options for doing HA storage including DRBD, clustered LVM2, and Cluster-MD. Cluster-MD is presented as a software-based RAID storage that provides redundancy at the device level across multiple nodes. Performance comparisons show Cluster-MD outperforming clustered LVM mirroring. Extensions to Cluster-MD are discussed including expanding the size of a Cluster-MD device.

Troubleshooting Redis- DaeMyung Kang, KakaoRedis Labs

Scylla on Kubernetes: Introducing the Scylla OperatorScyllaDB

The document introduces the Scylla Operator for Kubernetes, which provides a management layer for Scylla on Kubernetes. It addresses some limitations of using StatefulSets alone to run Scylla, such as safe scale down operations and tracking member identity. The operator implements the controller pattern with custom resources to deploy and manage Scylla clusters on Kubernetes. It handles tasks like cluster creation and scale up/down while addressing issues like local storage failures.

Choosing A Proxy Server - Apachecon 2014bryan_call

This document summarizes a presentation about choosing a proxy server. It discusses several popular proxy options including Apache Traffic Server (ATS), Nginx, Squid, Varnish, and Apache HTTP Server. It covers the types of proxies each supports, features, architectures, caching, performance, and pros and cons. Benchmark tests show ATS has the best cache scaling and performance overall while using less CPU than alternatives like Squid. Nginx and Squid had some issues with latency and HTTP compliance. The document recommends ATS as a good choice for its scaling, efficient caching, and plugin support.

Introduction to Apache KafkaShiao-An Yuan

This document provides an introduction to Apache Kafka. It describes Kafka as a distributed messaging system with features like durability, scalability, publish-subscribe capabilities, and ordering. It discusses key Kafka concepts like producers, consumers, topics, partitions and brokers. It also summarizes use cases for Kafka and how to implement producers and consumers in code. Finally, it briefly outlines related tools like Kafka Connect and Kafka Streams that build upon the Kafka platform.

VMworld Europe 2014: Virtual SAN Best Practices and Use CasesVMworld

This document provides an overview and agenda for a presentation on VMware Virtual SAN. It discusses key features of Virtual SAN including its software-defined storage approach and hybrid storage using SSD and HDD. Several use cases are reviewed like virtual desktop infrastructure, remote office/branch office, and DMZ/isolated environments. Best practices are also covered for various use cases around sizing, policies, and ready nodes. The document aims to introduce attendees to Virtual SAN capabilities and considerations for different deployment scenarios.

4Developers 2015: Scaling LAMP doesn't have to suck - Sebastian GrodzickiPROIDEA

Sebastian Grodzicki Language: Polish Tradycyjny LAMP sprawdza się świetnie … w środowisku deweloperskim. Nawet jeżeli nie tworzysz kolejnego Facebooka, to każda niedostępność Twojego serwisu kosztuje Cię sporo nerwów oraz pieniędzy. Awaria wystąpi wcześniej lub później. Pytanie nie brzmi "czy" leczy "kiedy". Dlatego warto o tym pomyśleć zawczasu i zbudować taką architekturę, która nie jest podatna na niedostępność z powodu awarii jednego z jej elementów. Jakie narzędzia wybrać? Jak wycisnąć z nich ostatnie soki?

Devopsconf2015 sebamontini-151117150231-lva1-app6891Flavia Marinelli

This document provides information about Amazon Web Services (AWS) instance types, pricing models, and best practices. It includes the following key points: - It describes the current AWS instance types for general purpose, compute optimized, memory optimized, GPU optimized, and high I/O workloads. - It outlines the small, medium, and large instance sizes and their characteristics like EBS optimization and network/storage performance. - It provides pricing details for the T2 instance family and Reserved Instances. - It compares the pricing of On Demand and Spot Instances for different EC2 instance types. - It discusses S3 and Glacier storage pricing and data retrieval costs. - It

Devopsconf 2015 sebamontiniSebastian Montini

This document provides information about Amazon Web Services (AWS) instance types, pricing models, and best practices. It includes the following key points: - It describes the current AWS instance types for general purpose, compute optimized, memory optimized, GPU optimized, and high I/O workloads. - It outlines the small, medium, and large instance sizes and notes that medium and large instances have EBS optimization and larger local storage. - It provides pricing and specifications for the T2 instance family and notes they are suited for development/testing workloads that are idle much of the time. - It briefly discusses reserved instances, spot instances, S3 and Glacier storage pricing, and common pitfalls around

Oracle RAC Presentation at Oracle Open WorldPaul Marden

Tweaking performance on high-load projectsDmitriy Dumanskiy

This document discusses optimizing performance for high-load projects. It summarizes the delivery loads and technologies used for several projects including mGage, mobclix and XXXX. It then discusses optimizations made to improve performance, including using Solr for search, Redis for real-time data, Hadoop for reporting, and various Java optimizations in moving to Java 7. Specific optimizations discussed include reducing garbage collection, improving random number generation, and minimizing I/O operations.

How Ceph performs on ARM Microserver ClusterAaron Joue

Dockersh and a brief intro to the docker internalsTomas Doran

Dockersh is a new tool to give a login shell into per-user Docker containers. (https://github.com/Yelp/dockersh) This talk will be an illustrated tour of what dockersh does, and why it might be useful to you. During this journey we’ll dive into the Go programming language, + libcontainer (the technologies Docker is built on) in addition to the facilities Docker uses in the kernel (Namespaces, Cgroups and Capabilities), how these work, and how normal mortals can (ab)use them for fun and profit

Docker in the WildChristian Mague

Docker in the WildChris Mague

Redis Meetup TLV - K8s Session 28/10/2018Danni Moiseyev

IMCSummit 2015 - Day 2 IT Business Track - 4 Myths about In-Memory Databases ...In-Memory Computing Summit

[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화OpenStack Korea Community

Orchestrating Redis & K8s OperatorsDoiT International

Docker vs kvmWilson Cunalata

SUSE Expert Days Paris 2018 - SUSE HA Cluster Multi-DeviceSUSE

Troubleshooting Redis- DaeMyung Kang, KakaoRedis Labs

Scylla on Kubernetes: Introducing the Scylla OperatorScyllaDB

Choosing A Proxy Server - Apachecon 2014bryan_call

Introduction to Apache KafkaShiao-An Yuan

VMworld Europe 2014: Virtual SAN Best Practices and Use CasesVMworld

4Developers 2015: Scaling LAMP doesn't have to suck - Sebastian GrodzickiPROIDEA

Devopsconf2015 sebamontini-151117150231-lva1-app6891Flavia Marinelli

Devopsconf 2015 sebamontiniSebastian Montini

Oracle RAC Presentation at Oracle Open WorldPaul Marden

Tweaking performance on high-load projectsDmitriy Dumanskiy

How Ceph performs on ARM Microserver ClusterAaron Joue

Dockersh and a brief intro to the docker internalsTomas Doran

More from Redis Labs (20)

Redis Day Bangalore 2020 - Session state caching with redisRedis Labs

This document discusses using Redis caching to improve performance for the DBS Paylah mobile wallet application. Paylah aims to significantly increase its user base which will increase load on its backend systems. Caching application data and session state in Redis can reduce latency, improve responsiveness for users, and reduce costs by lowering load on legacy backend databases and mainframes. The document outlines some key Paylah use cases where caching transaction histories and account details in Redis would accelerate retrieval and improve the mobile experience by avoiding the need to access slower backend systems on each request.

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020Redis Labs

The document discusses rate limiting and metering using Redis. It begins by introducing rate limiting and metering and why Redis is well-suited for these tasks. It then covers different Redis data structures that can be used, such as lists, hashes, sorted sets and strings. Common Redis commands for counting, setting keys and checking time to live are also presented. Different rate limiting design patterns and anti-patterns are described, including fixed window, sliding window and token bucket approaches. Finally, resources for further information are provided.

The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...Redis Labs

The document summarizes a presentation about using Protocol Buffers and Redis together. It discusses how Protocol Buffers provide strict data types, versioning, and serialization/deserialization benefits. It then outlines Redis key patterns using namespaces, versions, data categories and identifiers. Examples are provided to show how Protocol Buffers messages can be stored in Redis using these key patterns, including storing connections data in sets, sorted sets and individual messages. Benefits discussed include structure, readability, testing and abstraction.

SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020Redis Labs

The document discusses common use cases for combining SQL, Redis, and Kubernetes including caching, session management, rate limiting, and data ingestion. It outlines how Kubernetes can be used for scaling microservices while Redis is used for data service scaling. The presentation proposes combining Redis, SQL Server, and Kubernetes with a proxy service, and describes using Redis for caching, session storage, and rate limiting of SQL data. It also discusses running Redis and front-end apps on Kubernetes and deploying SQL as a Kubernetes service through a proxy.

Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...Redis Labs

This document discusses using Rust and Redis to build cloud native platforms. It first provides context about devops and the need to do more with less. It then discusses how platforms are becoming more distributed and Kubernetes upends distribution paradigms. The document dives into how Rust addresses issues like concurrency and systems programming. It also discusses how Redis can be used for caching, queues, streams and more. Finally, it mentions that Rust and Redis will be demonstrated.

Redis for Data Science and Engineering by Dmitry Polyakovsky of OracleRedis Labs

This document contains a presentation about using Redis for data science and engineering. It introduces the presenter and provides an agenda that covers using Redis for data science and data engineering. The presentation notes that Redis can be used as both a data store and job queue, has flexible data structures and is fast, though it uses RAM and cannot query by value. It also lists Python Pandas and includes a demo and links for further information.

Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020Redis Labs

Jamie Scott from RedisLabs presented on practical use cases for access control lists (ACLs) in Redis 6. The presentation covered new security features in Redis 6 including encryption in transit, key space and command restrictions, and multiple access control list users. It demonstrated how ACLs allow users to define access based on key labels and restrictions. ACLs can facilitate discretionary and mandatory access controls. The presentation showed examples of using ACLs to restrict user access by key labels and commands to enhance operational security.

Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020Redis Labs

This document summarizes a presentation about Redis version 6 and beyond. Some key points include: - Redis version 6 includes new features like ACL for security, client-side caching, diskless replication, and multi-threaded I/O. - Redis is positioned as both a cache and a database due to its speed, data structures, and ability to handle complex data models through modules. - Redis Enterprise provides additional capabilities like durability, high availability, geo-distribution, security and multi-tenancy. - Modern data models in Redis modules include Streams, RediSearch, RedisGraph, RedisTimeSeries, RedisAI, RedisJSON and RedisBloom. - RedisInsight is

Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...Redis Labs

The document discusses how Sinclair Broadcast Group leverages Redis for system monitoring of its content delivery network. It operates 193 news stations with 10,000 active pages daily and millions in archive. New stories are posted every 15 seconds and must be visible across its 1,000+ targets within 1 minute. Redis is used to track performance across the multi-level CDN and ensure service level agreements are met with real-time resolution and alerting. It provides a black box view of the audience experience and can scale monitoring to all relevant pages within 30 seconds. Redis acts as a distributed data store to parallelize the monitoring task across the large scale of the network.

JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...Redis Labs

The document summarizes a presentation about when to use the RedisJSON data type. It discusses how Coupang uses Redis extensively for their ad platform. It then compares the performance and memory usage of storing JSON data as strings, hashes, or using the RedisJSON data type. Benchmark results show RedisJSON can provide better performance for retrieving and updating JSON fields compared to strings and hashes, though it uses more memory. The document recommends using RedisJSON for smaller JSON payloads after benchmarking and memory monitoring.

Highly Available Persistent Session Management Service by Mohamed Elmergawi o...Redis Labs

The document discusses the challenges of building a highly available persistent session management service. It describes Zulily's legacy architecture which lacked high availability and required manual intervention. A new architecture is proposed using Redis for persistent storage, Dynomite for real-time replication across data centers, and a connection pooling proxy to improve efficiency and distribute load. The architecture provides high availability through replication, reduces overhead through connection pooling, and handles failures through consistent hashing and health checks. It was tested through simulations and showed a failure rate of only 0.42% during outages.

Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...Redis Labs

The document describes the process that a Redis command follows from the client side to the server side. On the client side, the command is sent over the network to the Redis server. On the server side, the command is read from the kernel buffers, validated, executed by calling the relevant command handler, and the response is written back to the client over the network. The core functions involved on the server side are ReadQueryFromClient(), ProcessInputBuffer(), ProcessCommand(), Call(), and handleClientsWithPendingWrites(). Redis 6.0 introduced I/O threads to handle reads and writes in parallel for improved performance while still maintaining Redis' single-threaded processing model.

Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...Redis Labs

This document discusses MDmetrix, a healthcare data intelligence company that uses RedisGraph to provide flexible analysis of hospital data. RedisGraph is a graph database that represents data as nodes and relationships and uses an adjacency matrix and linear algebra to query the graph. MDmetrix models its healthcare data as a property graph in RedisGraph to allow for complex queries across different data dimensions like patients, facilities, procedures and drugs. RedisGraph allows MDmetrix to query the data more easily than traditional OLAP cubes or relational databases due to the semi-structured and flexible nature of the graph model.

RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020Redis Labs

RediSearch 1.6 includes a new low-level API that allows other Redis modules to embed RediSearch indexing capabilities. It also introduces index aliasing and several performance improvements such as forked thread garbage collection. Based on benchmarks, RediSearch 1.6 shows 48-73% better performance than version 1.4, particularly during high update rates where it maintains more stable read latencies.

RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020Redis Labs

RedisGraph 2.0 provides significant improvements including: - Full text search support through embedded RediSearch 1.6 enabling graph-aided search. - Support for returning full graph responses to enable better visualization. - Broad support for Cypher including triadic closure and new graph-aided search capabilities. - Performance improvements of up to 3.7x faster operations per second and 3.9x faster query times compared to RedisGraph v1.2. - Support for benchmarking including the LDBC benchmark.

RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020Redis Labs

RedisTimeSeries is a time-series database that provides compression to reduce memory usage by up to 98% and improve performance. The RedisTimeSeries 1.2 release includes compression algorithms based on a Facebook paper that provide stable ingestion times independent of the number of data points. It also includes a reviewed API with performance improvements and clearer functionality. Performance testing showed ingestion throughput improved by 2-3% and query performance increased from 15-70% with the new release compared to the previous version.

RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020Redis Labs

This document summarizes RedisAI 0.9 and its capabilities for model deployment and benchmarking. It introduces RedisAI's new tensor data type and ability to deploy models to CPU and GPU. It then discusses AIBench, a tool developed to benchmark AI serving solutions like RedisAI, TensorFlow Serving, and REST APIs. The benchmarks show RedisAI providing 5.5x and 2.5x more inferences than REST APIs and TensorFlow Serving respectively, due to its data locality. The document concludes by mentioning RedisAI's integration with MLFlow for model deployment with a single command.

Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...Redis Labs

The document discusses how Freshworks uses Redis Labs to rate limit 30 million API requests per day through their API gateway called Fluffy. Fluffy stores rate limit policies and maintains counters to track requests. Redis Labs allows Fluffy to easily scale to handle the high volume of requests by providing a fast, in-memory data store for managing rate limiting counters. The system was able to successfully rate limit 30 million requests per day with Redis Labs.

Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...Redis Labs

Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...Redis Labs

Redis was used by Myntra to solve several complex scaling problems. It was used to build a scalable user segment service to support high read throughput of up to 5 million requests per minute with low latency. Redis allowed the service to scale beyond a single instance and included features like automatic backups and memory management. Redis also helped build a scalable mobile verification platform to reliably handle 100,000 requests per minute and scale to support higher future volumes. It was used as both a transient store and persistent backend. Finally, Redis locks helped build a scalable A/B testing platform by allowing experiments to be created and updated in an orderly concurrent fashion.