Counting image views using redis cluster

Jul 6, 20170 likes3,438 views

Streaming Logs and Processing View Counts using Redis Cluster Seandon Mooy (Imgur) When you browse through Imgur, you notice that each user's post includes the number of views for that particular post. Imgur processes over 3 billion views per month and powers our view count feature using Redis. In this talk, we cover our current architecture for streaming logs and processing view counts using Redis Cluster, as well as some of the alternatives we explored and why we chose Redis.

Counting Image Views using Redis Cluster
Seandon Mooy
DevOps Engineer
@erulabs

Counting Image Views using Redis Cluster
Or…. how I stopped map-reducing and learned to love the stream
Seandon Mooy
DevOps Engineer
@erulabs

Counting image views using redis cluster

Challenges with Hbase
Roughly 5% of all requests
through THRIFT were
failing… So many tunables!

Challenges with Hbase
Roughly 5% of all requests
through THRIFT were
failing… So many tunables!
Optimized timeouts,
added circuitbreakers, etc
Trickle of working requests
during outage means circuit
breakers are hard to design…

Fastly
ViewCount V2 - Real time with less complexity!
TCP syslog stream

Fastly
ViewCount V2 - Real time with less complexity!
TCP syslog stream
Ingest service

Fastly
ViewCount V2 - Real time with less complexity!
TCP syslog stream
Ingest service
Parses syslog lines, reports
metrics via statsd

Fastly
ViewCount V2 - Real time with less complexity!
TCP syslog stream
Ingest service
Parses syslog lines, reports
metrics via statsd
Redis 3.2 cluster!

Fastly
ViewCount V2 - Real time with less complexity!
Ingest service
Hbase Backfill service

Fastly
ViewCount V2 - Real time with less complexity!
Ingest service
Hbase Backfill service
Internet
API service

ViewCount V2 - Results:
Request latency:
min: 1ms
max: 16.9ms
median: 1.6ms
p95: 2.6ms
p99: 4.6ms
Codes:
200: 10000

ViewCount V2 - Results:
20 billion commands!
> 400GB in memory!

Things to be aware of:
1. Redis Cluster shard maps - redirections, etc.
Monitor redirections - gracefully restart workers after shard moves
2. AOF can slow down / fail large “redis-trib.rb” operations.
Make sure to disable before / re-enable after!
3. Not all legacy systems support Redis Cluster, and if they do…
They might not support it well (PHP-FPM)!
4. Over memory capacity behavior?
Previously we would hard-crash - now we’d LRU old 1-view images.
Neither are good, but for us, one is much less painful

ViewCount V3?
Approaching the point of minimal gains for man-hours, but what else might be fun?
1. Moving PHP7 off NodeJS API and directly to Redis Cluster
Downsides: dealing with shard maps is complex is a stateless / process-per-request environment!
2. Using redis3's BITFIELD or HSet to save on key storage costs
Downsides: complicate the system, reduce “hit-by-a-bus” issues - keys are just hashes, values are just counts!
3. Dealing with the nature of TCP Streams (TCP is not HTTP!)
One connection to rule them all! - Node’s Cluster module helps,
but perhaps Rust or Golang?
Downsides: Vertical scaling is non-obvious on EC2

ViewCount V2 - Results:
Redis is:
Faster - Imgur response time decreased ~50ms

ViewCount V2 - Results:
Redis is:
Faster - Imgur response time decreased ~50ms
Cheaper - EC2 cost reduced by 75%

ViewCount V2 - Results:
Redis is:
Faster - Imgur response time decreased ~50ms
Cheaper - EC2 cost reduced by 75%
Simpler - No Java, no MR, no ZK, no third parties, just INCR + GET!

Redis is:
Faster - Imgur response time decreased ~50ms
Cheaper - EC2 cost reduced by 75%
Simpler - No Java, no MR, no ZK, no third parties, just INCR + GET!
More fun! - I got to talk at RedisConf17!
ViewCount V2 - Results:

Acknowledgment
Imgur DevOps Team
Imgur Platform Team

The document discusses Nighthawk, Twitter's distributed caching system which uses Redis. It provides caching services at a massive scale of over 10 million queries per second and 10 terabytes of data across 3000 Redis nodes. The key aspects of Nighthawk's architecture that allow it to scale are its use of a client-oblivious proxy layer and cluster manager that can independently scale and rebalance partitions across Redis nodes. It also employs replication between data centers to provide high availability even in the event of node failures. Some challenges discussed are handling "hot keys" that get an unusually high volume of requests and more efficiently warming up replicas when nodes fail.

Troubleshooting redisDaeMyung Kang

This document discusses troubleshooting Redis. Some key points: - Redis is single-threaded, so commands like KEYS, FlushAll, and deleting large collections can be slow. It's better to use SCAN instead of KEYS. - Creating Redis database snapshots (RDB files) and rewriting the append-only file (AOF) can cause high disk I/O and CPU usage. It's best to disable automatic rewrites. - Monitoring memory usage and fragmentation is important to avoid performance issues. The maxmemory setting also needs monitoring to prevent out-of-memory errors. - Network and replication failures need solutions like DNS failover or using Zookeeper for coordination to maintain high availability of Redis

Day 2 General Session Presentations RedisConfRedis Labs

The document discusses new memory technologies like persistent memory and their implications. It provides latency and bandwidth numbers for different memory types and notes that heterogeneous memory systems using tiers of DRAM and NVM provide opportunities for better performance and cost. Examples are given of key-value stores and databases leveraging NVM to achieve high performance while reducing costs. The talk also discusses how new distributed data structures like CRDTs could be used across servers with shared memory.

RedisConf17 - Redis in High Traffic Adtech StackRedis Labs

Rahul Babbar presented on Times Internet's use of Redis in their high-traffic adtech stack. Some key points: - Times Internet is India's largest digital company with 240+ million unique visitors per month. - Their adtech stack powers ads for 150+ publishers and handles 9 billion ad impressions per month. - Redis is used for caching, user profiles, analytics and more due to its low latency and ability to handle high volumes. - Their Redis clusters consist of over 160 nodes across 4 clusters, serving over 2 million queries per second with 99% under 2ms response times.

Using Redis at FacebookRedis Labs

The document discusses how Redis is used at Facebook to solve two problems: 1) Serving user configurations in real-time by using a hierarchical master-slave replication architecture with global and regional masters and slaves to distribute reads and allow writes at different levels. 2) Processing stats data in real-time by using a split data processing model where daily bulk processing is done separately from real-time processing by tailers that insert stats into a sharded Redis cluster using HINCRBY.

RedisConf17 - Lyft - Geospatial at Scale - Daniel HochmanRedis Labs

This document discusses Lyft's migration from a monolithic architecture to using Redis to power a geospatial indexing system at scale. It describes Lyft's original architecture, issues they faced, and how they iterated on their data model and use of Redis over time. It also discusses how Lyft uses Redis across their platform, including operations, monitoring, capacity planning, and their work contributing to open source projects like Envoy.

What's new with enterprise Redis - Leena Joshi, Redis LabsRedis Labs

Redis Labs manages over 160k+ HA databases, 10k clustered databases, without data loss in spite of one node failure a day and one data center outage per month. Using Enterprise Redis(RLEC), Redis Labs delivers seamless zero downtime scaling, true high availability with persistence, cross-rack/zone/ datacenter replication and instant automatic failover. Learn how. Join this session for a deep dive into how enterprise Redis makes for no-hassle Redis deployments and the roadmap for new Redis capabilities. Discover new cost savings with Redis on Flash for cost-effective high performance operations and analytics

Handling Redis failover with ZooKeeperryanlecompte

This document discusses using ZooKeeper to automatically handle Redis failover. ZooKeeper is an open-source tool that provides primitives for building distributed applications and handles tasks like leader election and quorum management. The presenter describes how his redis_failover Ruby gem uses ZooKeeper to monitor Redis servers, detect failures, and automatically inform clients so they reconnect to the new master, preventing downtime during a failover. Several companies already use this approach with redis_failover to make their Redis infrastructure more robust and fault-tolerant.

Redis for horizontally scaled data processing at jFrog bintrayRedis Labs

Redis Day Keynote Salvatore Sanfillipo Redis LabsRedis Labs

Redis Developers Day 2014 - Redis Labs TalksRedis Labs

RedisConf17 - Redis Cluster at flickr and tripodRedis Labs

Flickr uses Redis extensively across many services for caching, queueing, and more. Some key uses of Redis include caching for the camera roll and activity feed, which each have multiple masters and slaves distributed across data centers. Notifications are also queued in Redis. In total, Flickr operates over 100 physical hosts running 500+ Redis instances to support over 600,000 operations per second across services like the camera roll, activity feed, notifications, and more.

Redis in a Multi Tenant Environment–High Availability, Monitoring & Much More! Redis Labs

This document discusses best practices for running Redis in a multi-tenant environment. It covers architectural considerations like high availability, security and isolation techniques using ACLs and SSL, and the importance of monitoring and understanding your environment. The key opportunities are that failover is easy to implement, changes can be introduced smoothly, and the architecture is reusable. Challenges include managing global sentinels and Redis drivers. Case studies demonstrate issues like customer spikes causing problems and the importance of monitoring.

Redis ReplicationIsmaeel Enjreny

Redis replication allows slave servers to mirror the data and state of a master server. It supports asynchronous replication where slaves acknowledge processed data from the master. A master can have multiple slaves in a graph-like structure. Replication is automatic and helps with scalability, high availability, and avoiding costly master disk writes. Partial resynchronization allows replication to resume after a temporary disruption without a full resync.

Perforce BTrees: The Arcane and the ProfanePerforce

Managing Redis with Kubernetes - Kelsey Hightower, GoogleRedis Labs

Managing 50K+ Redis Databases Over 4 Public Clouds ... with a Tiny Devops TeamRedis Labs

Redis is an in-memory database that provides fast performance for powering lightning fast apps. It supports many data structures like strings, hashes, lists, sets and sorted sets. Redis is efficient due to its support for many data structures and commands, as well as its complexity-aware design. Redis Labs provides fully-managed cloud services for Redis and Memcached, and helps customers with challenges around scalability, high availability, performance and monitoring for large-scale Redis deployments.

Dynomite: A Highly Available, Distributed and Scalable Dynamo Layer--Ioannis ...Redis Labs

Dynomite is a thin, distributed dynamo layer for different storage engines and protocols. Currently at Netflix, we are focusing on using Redis as the storage engine. Dynomite supports multi-datacenter replication and is designed for high availability. In the age of high scalability and big data, Dynomite’s design goal is to turn single-server datastore solutions into peer-to-peer, linearly scalable, clustered systems while still preserving the native client/server protocols of the datastores, e.g., Redis protocol. In this talk, we are going to present Dynomite recent features, and the Dyno client. Both projects are open source and available to the community.

RADOS improvements and roadmap - Greg Farnum, Josh Durgin, Kefu ChaiCeph Community

Scylla Summit 2022: ScyllaDB Rust Driver: One Driver to Rule Them AllScyllaDB

The idea for implementing a brand new Rust driver for ScyllaDB emerged from an internal hackathon in 2020. The initial goal was to provide a native implementation of a CQL driver, fully compatible with Apache Cassandra™, but also contain a variety of Scylla-specific optimizations. The development was later continued as a Warsaw University project led by ScyllaDB. Now it's an officially supported driver with excellent performance and a wide range of features. This session shares the design decisions taken in implementing the driver and its roadmap. It also presents a forward-thinking plan to unify other Scylla-specific drivers by translating them to bindings to our Rust driver, using work on our C++ driver as an example. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Ceph Client librbd Performance Analysis and Learnings - Mahati ChamarthyCeph Community

Highly scalable caching service on cloud - RedisKrishna-Kumar

Leveraging Structured Data To Reduce Disk, IO & Network BandwidthPerforce

Experiences building a distributed shared log on RADOS - Noah WatkinsCeph Community

This document summarizes Noah Watkins' presentation on building a distributed shared log using Ceph. The key points are: 1) Noah discusses how shared logs are challenging to scale due to the need to funnel all writes through a total ordering engine. This bottlenecks performance. 2) CORFU is introduced as a shared log design that decouples I/O from ordering by striping the log across flash devices and using a sequencer to assign positions. 3) Noah then explains how the components of CORFU can be mapped onto Ceph, using RADOS object classes, librados, and striping policies to implement the shared log without requiring custom hardware interfaces. 4) ZLog is presented

Linux Block Cache Practice on Ceph BlueStore - Junxin ZhangCeph Community

This document discusses using Linux block caching with Ceph BlueStore. It explains that BlueStore can better utilize fast storage devices like SSDs compared to FileStore. It tested using Bcache and DM-writeboost to cache BlueStore data on HDDs using SSDs. Bcache performed better overall. Issues found were slow requests when caching and BlueStore used the same SSD, and inconsistency in SSD data management between BlueStore and the cache. Future work could have BlueStore control all raw disks and prioritize data saving to fast devices.

RedisConf17 - Redis Development, An Update - @antirezRedis Labs

Redis 4.0 will include several major new features and improvements including PSYNC2 for better replication, UNLINK for non-blocking DEL operations, and modules to allow building additional functionality on Redis. Streams will be a new data type in Redis 4.2, providing an abstract log file structure for ordered, timestamped entries that allows for efficient random access, deletion of old data, and multiple consumers similar to Kafka. Streams are designed to address use cases like time series data and message queues more efficiently than existing Redis data types.

Ceph Goes on Online at Qihoo 360 - Xuehan XuCeph Community

The document summarizes Qihoo 360's experience deploying Ceph for storage at scale. They use Ceph RBD for virtual machine images and CephFS for a shared file system. For Ceph RBD, they have over 500 nodes across 30+ clusters storing over 1000 object storage devices. They use both full SSD and hybrid SSD/HDD clusters depending on performance needs. Their experience highlights best practices for deployment, performance, stability and operations. For CephFS, they evaluated metadata performance and discussed considerations for a production deployment.

Red Hat Storage RoadmapColleen Corrice

Red Hat's storage roadmap aims to offer a unified, open software-defined storage portfolio for next generation workloads. Their portfolio includes Ceph and Gluster storage, which provide data services like object storage, block storage, and file sharing on standard hardware. Red Hat is working to improve management, performance, security, and integration of these products. They are also exploring new workloads like running MySQL on Ceph block storage and using Gluster Storage for hyperconverged Red Hat Enterprise Virtualization environments.

Compressed Introduction to Hadoop, SQL-on-Hadoop and NoSQLArseny Chernov

Infrastructure as code might be literally impossible part 2ice799

The document discusses various issues with infrastructure as code including complexities that arise from software licenses, bugs, and inconsistencies across tools and platforms. Specific examples covered include problems with SSL and APT package management on Debian/Ubuntu, Linux networking configuration difficulties, and inconsistencies in Python packaging related to naming conventions for packages containing hyphens, underscores, or periods. Potential causes discussed include legacy code, lack of time for thorough testing and bug fixing, and economic pressures against developing fully working software systems.

More Related Content

What's hot (20)