Replication and Consistency in Cassandra... What Does it All Mean? (Christopher Bradford, DataStax) | C* Summit 2016

Editor's Notes

• #4: I have contributed to multiple open source projects around distributed computing
• #5: Today we are talking about Consistency & Replication. We want to dive into how are these key components are implemented in Cassandra and what do they mean to Developers and Operations?
• #6: We’ll start with the CAP theorem. The CAP theorem says that it is impossible for a distributed system to provide guarantees around the following areas. Autumn of 1998
• #7: Consistency - Every read receives the most recent write or an error
• #8: Availability - Every request receives a response Note the data returned is not guaranteed to be the most recent
• #9: Wikipedia says “due to network failures”, given my week it is due to disk controller errors. Partition Tolerance – the system continues despite arbitrary partitioning
• #10: Eric Brewer, creator of the CAP theorem, argues that the pick 2 statement was a bit misleading. Really system designers need to choose where to be flexible in the case of a partition. Some systems prefer availability while Cassandra focuses on consistency.
• #11: Consistency may be flexible to meet the needs of the application!
• #12: How are availability and partition tolerance achieved? Through replication
• #13: Let’s look at how replication affects availability and partition tolerance. Assume in this environment we have a replication factor of 3. This means every piece of data written to our cluster is stored 3 times. The client issues a write to a node in the cluster. This node is known as the coordinator for the duration of this request. The coordinator hashes the PARTITION KEY portion of the PRIMARY KEY and uses a REPLICATION STRATEGY to determine where the replicas live and forwards the request to those nodes.
• #14: When one of the replicas is down the coordinator still writes to the other two replicas. This satisfies the availability portion of the CAP Theorem A hint is then stored on the coordinator. Should the replica that was down come back up within a certain time period (default 3 hours) the hint will be replayed against that host. This helps get the node back in sync with the rest of the cluster. This satisfies the partition tolerant portion of the CAP theorem
• #15: Reads behave in a similar manner (although without the need for a hint). The coordinator is aware that one of the replicas is down and responds with data from the two replicas that are still available.
• #16: Replication is defined at the keyspace level. The number of replicas and placement strategy are supplied during keyspace creation. The replication strategy is specified with the class parameter. Other parameters may also be required, but they are tied to the strategy being used.
• #17: Cassandra ships with a few replication strategies. Let’s look at those now.
• #18: The Simple Strategy is just that, Simple. The replicas for a particular piece of data circle the ring in order.
• #19: If the token hashes differently then it may look like this. WARNING: Be very careful with SimpleStrategy in multi-DC environment. Depending on the consistency level and load balancing policy writes may throw exceptions as hosts are not necessarily routable.
• #20: Network Topology Strategy layers additional logic into the replica selection process. It utilizes topology information to place replicas in a manner which ensures an even higher level of reliability. In our first example let’s assume every node is in the same DC and physical rack. The replica selection process looks similar to the SimpleStrategy, there is nothing too intense here. Note you must use an appropriate Snitch. The snitch defines how topology information is shared in the cluster. The GossipingPropertyFileSnitch uses the cassandra-rackdc.properties file to share topology info with peers where the SimpleSnitch has hardcoded values 
• #21: In this example we have two racks, the aptly named rack 1 and rack 2. Now the network topology strategy will try and protect against an entire rack failure by placing replicas on different racks. In fact it won’t even attempt to write to another node in the same rack until all other rack options have been exhausted.
• #22: In this example we have two racks added a third rack. Now one replica will be placed on each of the racks. This will lead to an imbalance in the cluster. The singular node in rack 1 will contain 100% of the data in the cluster.
• #23: Remember how we discussed how the CAP theorem has evolved over time. It’s clear in Cassandra that this is at the heart. Consistency can be tuned on a per-query level to enhance performance or ensure consistency.
• #24: We call this tunable consistency. Consistency can even be tuned depending on cluster health to ensure application availability. Let’s look at some of these levels.
• #25: The consistency level is used to determine the number of replicas that must respond for any given query. Some levels are unique to the type of query being performed (read vs write). Others span datacenters and are not expected to be extremely performant.
• #26: Discuss advantages of each, speed, correctness, failure tolerance.
• #27: Discuss advantages of each, speed, correctness, failure tolerance. Note that even though we may return after one replica acknowledged the write we still attempt the write on other replicas. Bring up immediate consistency.
• #28: An exception is thrown! At this point you can alert the user to a potential issue or try the request with a different consistency level. This process may be automated with a RetryPolicy. This is a big win for your application as you can stay available for users while alerting them that there may be a slight delay on changes.
• #29: Cassandra has a number of anti-entropy processes to ensure nodes respond with the latest information. We were first introduced to this earlier with hinted handoffs. Should hints expire a manual repair of the node will synchronize it with its neighbors. There is also a read repair system in place. This takes the data returned by multiple replicas, compares them, and returns the latest value. A write is also issued which fixes any replicas that are out of date. This was taken a step further by Netflix with the Cassandra tickler. This process performs a read at the ALL consistency level across every primary key value forcing replicas to be brought in sync.
• #30: Replication Controls where copies live Set on the keyspace level Are imperative both during a and p situations Consistency Dictates trade-offs between performance and correctness Achieves synchronization of replicas Consistency levels Both are core building blocks of Cassandra. Understand their usage and don’t be afraid to jump in the code. The classes that make this up are fairly simple and easy to reason about.