Cassandra Front Lines

Oct 13, 20150 likes1,228 views

Ben Bromhead is the co-founder and CTO of Instaclustr, which provides Cassandra-as-a-Service. Instaclustr manages 50+ Cassandra nodes for customers. Early on, Instaclustr encountered issues like a Cassandra bug causing assertion errors for large column names and had to perform an emergency migration for a customer whose self-managed cluster was down for 48 hours. Migrations and real-world usage revealed new challenges compared to initial perfect test scenarios.

From the front lines
Saving stranded clusters
#CassandraSummit

Who am I and what do I do?
• Ben Bromhead
• Co-founder and CTO of Instaclustr -> www.instaclustr.com
• Instaclustr provides Cassandra-as-a-Service in the cloud.
• Currently in AWS, Google Cloud in private beta with more to come.
• We currently manage 50+ nodes for various customers, who do
various things with it.

In the beginning…
• Well designed schemas
• No data migrations
• Everything was perfect and happy
• Then we got customers

Our ﬁrst C* patch
• CASSANDRA-6521
• Cassandra wouldn’t check the length of a column name in a range
predicate for slice operations.
• So for large column names it would throw an assertion error.
• Which would in turn tie up threads, causing the node to be
unresponsive and eventually the whole cluster.

Our ﬁrst C* patch
• What was the size of the column name that would cause this issue?
• Around 130kb
• wat…

Our ﬁrst migration
• Receive frantic phone call
• Self managed cluster has been down for 48 hours, for a company
that gets 25 million monthly unique views.
• They are hurting

Our ﬁrst migration
• The cluster was running a very early version of C* 2.0
• Update/patch the old cluster, get everything back online
• Start the migration process…

Our ﬁrst migration
• Bulkload manages to kill their new cluster with us in about 5
minutes.
• Open logs
• Read 1 live and 38456 tombstoned cells (see
tombstone_warn_threshold)
• For every column family
• wat…

Conclusion
• Everything is awesome
• Then reality occurs
• It’s actually way more fun
• Want to make C* even better? We are hiring!

1) Ben Bromhead is the CTO of Instaclustr, which provides Cassandra-as-a-Service. When adding capacity to an existing Cassandra cluster, joining nodes normally bootstrap by streaming data from existing nodes, adding load. 2) "Bootstrap from backups" is proposed as a solution where joining nodes stream data directly from backups stored in object storage rather than existing cluster nodes, reducing load on the cluster. 3) This allows more reactive scaling with fewer side effects than typical predictive capacity planning approaches, and makes clusters more cost effective to run. The technique is currently in beta testing.

Micro-batching: High-performance writesInstaclustr

- Micro-batching involves grouping statements into small batches to improve throughput and reduce network overhead when writing to Cassandra. - A benchmark was conducted to compare individual statements, regular batches, and partition-aware batches when inserting 1 million rows into Cassandra. - The results showed that partition-aware batches had shorter runtime, lower client and cluster CPU usage, and was more performant overall compared to individual statements and regular batches. However, it may have higher latency which is better suited for bulk data processing rather than real-time workloads.

Multi-Region Cassandra ClustersInstaclustr

This document discusses enabling multi-region Cassandra clusters that span heterogeneous data centers using Network Address Translation (NAT) and DNS-based Service Discovery (DNS-SD). It describes how NAT allows sharing a limited number of public IP addresses between private nodes by mapping private ports to public ports. DNS-SD is proposed to advertise the port mappings so nodes can discover each other, with SRV and TXT records storing port and cluster details. Minor modifications to Cassandra and drivers are suggested to lookup ports via DNS-SD during connection establishment.

mParticle's Journey to Scylla from CassandraScyllaDB

mParticle processes 50 billion monthly messages and needed a data store that provides full availability and performance. They previously used Cassandra but faced issues with high latency, complicated tuning, and backlogs of up to 20 hours. They tested Scylla and found it provided significantly lower latency and compaction backlogs with minimal tuning needed. Scylla also offered knowledgeable support. mParticle migrated their data from Cassandra to Scylla, which immediately kept up with their data loads with little to no backlog.

C* Summit 2013: Cassandra at eBay Scale by Feng Qu and Anurag JambhekarDataStax Academy

We have seen rapid adoption of C* at eBay in past two years. We have made tremendous efforts to integrate C* into existing database platforms, including Oracle, MySQL, Postgres, MongoDB, XMP etc.. We also scale C* to meet business requirement and encountered technical challenges you only see at eBay scale, 100TB data on hundreds of nodes. We will share our experience of deployment automation, managing, monitoring, reporting for both Apache Cassandra and DataStax enterprise.

Introducing DataStax Enterprise 4.7DataStax

Running Cassandra in AWSDataStax Academy

For this upcoming meetup, we welcome Patrick Eaton PhD, Systems Architect at Stackdriver, and Joey Imbasciano, Cloud Platform Engineer at Stackdriver. What You'll Learn At This Meetup: • Why Stackdriver chose Cassandra over other DB offerings • Stackdriver's data pipeline that runs into Cassandra • Operating Cassandra Running on AWS • Stackdriver's approach to disaster recovery Patrick and Joey will be presenting their use of Apache Cassandra at Stackdriver, some lesson's learned, technical tips and a Q&A to end the evening.

Cassandra Summit 2014: Apache Cassandra Best Practices at EbayDataStax Academy

Presenter: Feng Qu, Principal DBA at eBay Cassandra has been adopted widely at eBay in recent years and used by many end-user facing applications. I will introduce best practices we have built over the time around system design, capacity planning, deployment automation, monitoring integration, performance analysis and troubleshooting. I will also share our experience working with DataStax support to provide a highly available, highly scalable data store fitting into eBay infrastructure.

Cassandra on Docker @ Walmart LabsDataStax Academy

This document discusses using Docker containers to run Cassandra clusters at Walmart. It proposes transforming existing Cassandra hardware into containers to better utilize unused compute. It also suggests building new Cassandra clusters in containers and migrating old clusters to double capacity on existing hardware and save costs. Benchmark results show Docker containers outperforming virtual machines on OpenStack and Azure in terms of reads, writes, throughput and latency for an in-house application.

From PoCs to ProductionDataStax

The document discusses best practices for moving Cassandra pilots and proofs of concept (PoCs) to production. It recommends starting with defining queries, building out 5-8 pilots, and designing REST APIs first. When moving to production, considerations include infrastructure, testing, coding practices like using asynchronous execution, data modeling for queries and analytics, and application optimization techniques.

Managing (Schema) Migrations in CassandraDataStax Academy

I'm going to cover something which could be seen as essential for Cassandra but which hasn't gotten much attention in the Cassandra community and literature. It's schema migrations--how you go about pushing out and versioning changes to your keyspace and table definitions across environments. This is an area that has established solutions in the relational database world, with tools like Liquibase(http://www.liquibase.org/) and Flyway (http://flywaydb.org/) and in web frameworks like Rails and Grails. I'll explain the different types of migrations but then focus, for most of the talk, on schema migrations. I'll explain how schema migrations have been done in the Cassandra community and the roadblocks teams have faced trying to use Liquibase and Flyway to manage Cassandra migrations. Then I'll share an elegant, lightweight schema migrations system that we at GridPoint built on top of Flyway. I'll use our system as a context for discussing schema migration best practices for Cassandra and the various choices teams have for their migrations and table definitions, including when NOT to use a tool like Flyway. I'll also touch on the other types of migrations besides keyspace and table definitions that can be versioned and driven off source control.

Critical Attributes for a High-Performance, Low-Latency DatabaseScyllaDB

This document discusses the attributes of a high-performance, low-latency database like ScyllaDB. It begins with introductions and an overview of ScyllaDB. It then summarizes how hardware has evolved over 20 years with more cores, memory, and faster disks. ScyllaDB was redesigned from first principles to take advantage of modern hardware, using an asynchronous, shared-nothing architecture with one shard per core. This allows it to achieve significantly higher performance than Cassandra. The document shows benchmark results demonstrating ScyllaDB's lower latencies and ability to scale to higher throughput. It also discusses how ScyllaDB uses workload prioritization to manage different types of workloads.

Cassandra Core ConceptsDataStax Academy

The document discusses different data storage options for small, medium, and large datasets. It argues that relational databases do not scale well for large datasets due to limitations with replication, normalization, sharding, and high availability. The document then introduces Apache Cassandra as a fast, distributed, highly available, and linearly scalable database that addresses these limitations through its use of a hash ring architecture and tunable consistency levels. It describes Cassandra's key features including replication, compaction, and multi-datacenter support.

Building clouds with apache cloudstack apache roadshow 2018ShapeBlue

Talk given at Apache Roadshow, FOSS Backstage, Berlin, June 2018 Apache CloudStack is open source software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. This talk will give an introduction to the technology, its history and its architecture. It will look common use-cases (and some real production deployments) that are seen across both public and private cloud infrastructures and where CloudStack can be completed by other open source technologies. The talk will also compare and contrast Apache Cloudstack with other IaaS platforms and why he thinks that the technology, combined with the Apache governance model will see CloudStack become the de-facto open source cloud platform. He will run a live demo of the software and talk about ways that people can get involved in the Apache CloudStack project.

Running Cassandra on Amazon EC2Dave Gardner

How to Monitor and Size Workloads on AWS i3 instancesScyllaDB

There is a new class of machines in town! Amazon recently unveiled i3, a new class of machines targeted at I/O-intensive workloads. Scylla will officially support i3, and previews are already available. Join our webinar to learn how to build a state-of-the-art database solution. Presenters Glauber Costa and Eyal Gutkind will cover how to: - Determine which workloads can benefit from i3 instances - Ensure Scylla fully leverages the great resources in the i3 family - Effectively navigate the Scylla monitoring system and identify bottlenecks You'll also see a live demonstration with a dashboard featuring an i3 cluster with different data models and workloads.

Overview of DataStax OpsCenterDataStax

Cassandra Lunch #92: Securing Apache Cassandra - Managing Roles and PermissionsAnant Corporation

Network, hardware, data, and application level security strategies are discussed for securing Apache Cassandra platforms. It is emphasized that roles and permissions must be carefully designed to segment access for system administrators, developers, applications, and other users. Key recommendations include creating roles for different tenant, application, and system access levels with varying permission levels; using authentication and authorization; and implementing network, system, and encryption security best practices.

Data Stores @ NetflixVinay Kumar Chella

Vinay Chella presents on Cassandra architecture and scalability at Netflix. Some key points include: 1) Netflix uses Cassandra to store 98% of streaming data. Cassandra clusters are managed using Priam to handle backups, configuration, and cluster operations. 2) Challenges in maintaining Cassandra clusters at scale are addressed through tools like Eunomia for monitoring and predictive analysis. Mantis provides real-time health monitoring through streaming cluster data. 3) Cassandra clusters are deployed on AWS for resilience across instances, availability zones, regions, and cloud providers. Priam handles tasks like automated token assignment and backups to S3 for disaster recovery.

Cassandra Performance and Scalability on AWSAdrian Cockcroft

The impact of cloud NSBCon NY by Yves GoelevenParticular Software

The Promise and Perils of Encrypting Cassandra Data (Ameesh Divatia, Baffle, ...DataStax

Why do data breaches occur even though we protect data at rest and in flight? What if the Cassandra admin's credentials get compromised? Why is it so hard to make encryption work for real world applications? If I encrypt my customer's data, do I have to turn it over when the authorities come calling? The answer may be in keeping data encrypted. . .always, let the customer own the keys and make data breaches irrelevant. In this talk, Ameesh Divatia, Co-founder at Baffle.io, will talk about a way to encrypt individual fields in a Cassandra database while continuing to let them be available for CQL access. From deterministic to random algorithms, key management and integration into DataStax drivers, this talk will introduce attendees to the steps to follow in order to protect an existing Cassandra database with field-level granularity ensuring protection against data breaches. About the Speaker Ameesh Divatia President & CEO, Baffle, Inc. Ameesh is a serial entrepreneur with over 25 years of operating experience in storage, security and networking infrastructure. He specializes in conceiving and implementing startup business plans that create new product categories by leveraging innovation in existing markets to its adjacencies. He co-founded Baffle in May 2015 to address the challenge of preventing data breaches in cloud infrastructure.

iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...DataStax Academy

iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information. In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc. Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.

On-demand Continuous Integration with Jenkins, jclouds, and CloudStackke4qqq

This document discusses using Jenkins, jclouds, and CloudStack together to provide scalable continuous integration testing. It describes how the author transitioned from using dedicated Jenkins masters and VMs to dynamically provisioning build slaves from CloudStack on demand using jclouds. The key steps were building standard images with Packer, configuring the jclouds plugin in Jenkins to integrate with CloudStack as the cloud provider, and configuring jobs to spin up slaves when needed and terminate them after 30 minutes of idle time. This approach provides scalable testing resources that match the varying demand of their many projects.

Ruby Driver Explained: DataStax Webinar May 5th 2015DataStax

Apache Cassandra is the leading distributed database platform purpose built for the demands of today's modern Web, Mobile, and IOT applications. To support this growing demand, DataStax makes it a priority to deliver drivers to support today's most common and cutting edge languages. Join Bulat Shakirzyanov, Software Engineer at DataStax, as he dives into the inner-workings of the DataStax Ruby Driver for Apache Cassandra. He will: • Provide a brief overview of Apache Cassandra's architecture and various terminology • Walk through the basics for installing and using the DataStax Ruby Driver • Drill into exactly why the Ruby Driver is fully asynchronous, both from the IO implementation as well as the Cassandra native protocol design • Look at how the driver uses load balancing policies as well as use cases for various built-in load balancing policies, such as token aware and data center aware round-robin • Discuss in detail the driver's error handling and fault tolerance and discuss various expected failure modes and how to handle them • Explain the importance of address resolution policies and take a look at those that ship with the driver

Azure + DataStax Enterprise Powers Office 365 Per User StoreDataStax Academy

Case Study: Troubleshooting Cassandra performance issues as a developerCarlos Alonso Pérez

CloudOpen Japan - Controlling the cost of your first cloudTim Mackey

As presented at CloudOpen Japan in Tokyo in 2015. Today everyone is talking about clouds, and some are building them, but far fewer are operating successful clouds. In this session we'll examine a variety of paradigm shifts must IT make when moving from a traditional virtualization and management mindset to operating a successful cloud. For most organizations, without careful planning the hype of a cloud solution can quickly overcome its capabilities and existing best practices can combine to create the worst possible cloud scenario -- a cloud which isn't economical to operate, and which is more cumbersome to manage than a traditional virtualization farm. Key topics covered will include; transitioning the operational paradigm, the impact of VM density on operations and network management, and preventing storage cost from outpacing requirements.

Cassandra-as-a-ServiceInstaclustr

This document discusses how database-as-a-service (DBaaS) using Cassandra is growing rapidly. It notes that DBaaS is forecast to be the fastest growing category over the next four years. It then provides best practices for using Cassandra on DBaaS, including starting small on non-critical applications, testing data modeling without premature tuning, and allowing the DBaaS to automatically scale across multiple regions, data centers, and providers.

Cassandra Bootstap from BackupsInstaclustr

More Related Content

What's hot (20)