Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016

Operations, consistency and failover
for multi DC clusters

CASSANDRA SUMMIT - SEPTEMBER 2016
Alexander Dejanovski
@alexanderdeja
Consultant
www.thelastpickle.com
Datastax MVP for Apache Cassandra
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

AboutThe Last Pickle 
We help people deliver and improve Apache
Cassandra based solutions.
With staff in 5 countries and over 50 years
combined experience in Apache Cassandra.

DC = DataCenter

Why multi DC ?
Consistency
Operations
Failover

Multi DC use cases 
Load Balancing
&
Disaster recovery

 
Geographical colocation
with clients

Separate operational
and
analytical workloads

Clusters consistency 
Strongly consistent clusters 
Low latency between DCs
and At least 3 DCs
and No search/analytical DC

Clusters consistency 
Eventually consistent clusters 
High latency between DCs
or exactly 2 DCs
or at least 1 search/analytical DC

Replication & consistency 
Create a keyspace for a single DC cluster :
CREATE KEYSPACE ks1 
WITH replication =
{ 
'class':'SimpleStrategy',
‘replication_factor' : 7
};

SimpleStrategy on a multi DC cluster

Create a keyspace for a multi DC cluster :
CREATE KEYSPACE ks1 
WITH replication =
{'class':'NetworkTopologyStrategy',
'dc1' : 3,
'dc2' : 3
};

NetworkTopologyStrategy on a multi DC cluster

Conﬁguring DC on nodes
 
With GossipingPropertyFileSnitch :  
 
conf/cassandra-rackdc.properties 
dc=DC2

Non DC-aware Consistency Levels 
ONE (default)
(TWO,THREE)
QUORUM
ALL

DC-aware Consistency Levels 
LOCAL_ONE (default)
LOCAL_QUORUM
EACH_QUORUM

Read & write path 
QUORUM WRITE on DC1 

LOCAL_QUORUM WRITE on DC1 

QUORUM READ on DC2 

LOCAL_QUORUM READ on DC1 

LOCAL_QUORUM READ on DC2 

Operations & configuration 
 
Specific configurations
for multi DC clusters
in conf/cassandra.yaml

 
Speciﬁc throttling for inter DC streaming
throughput :
inter_dc_stream_throughput_outbound_megabits_per_sec
Defaults to 200 Mbps ( 25 MB/s )

 
Internode encryption can be activated for
inter DC communications only
server_encryption_options:
internode_encryption: dc

 
Internode compression can be activated for
inter DC communications only
internode_compression: dc

 
Reduce the TCP overhead in async DCs by setting :
inter_dc_tcp_nodelay: true
Larger but fewer TCP packets

 
Adding a new DC to an existing cluster

Operations - adding a new DC 
 
Migrate all your SimpleStrategy KS to
NetworkTopologyStrategy

 
ALTER KEYSPACE ks1 
WITH replication =
'dc1' : 3
};

 
Disable auto bootstrap on new nodes

 
Disable auto bootstrap on new nodes
(not mandatory, but safer…)

 
Add this in conf/cassandra.yaml : 
auto_bootstrap: false

 
Start new nodes

At this point, Nodes in the new DC are : 
• empty
• not involved in reads nor writes

 
Change strategy params to add replicas on
the new DC

 
You might want to make sure
trafﬁc is restricted to DC1
before you move on…
(unless you’re using QUORUM)

 
WITH replication =
'dc1' : 3,
'dc2' : 3
};

 
At this point, your new DC
accepts both reads and writes

 
But nodes on the new DC
are still desperately empty

 
Routing trafﬁc on a speciﬁc DC
is a dev task

 
1/ Pick a coordinator in a speciﬁc DC :
use DCAwareRoundRobinPolicy in your
<paste your language> Datastax driver

 
2/ tell the coordinator to work with nodes
in its own DC only :
use a LOCAL_* CL

Consistency level can be modified on the fly
through feature flips for example
Load balancing policy cannot…

 
Fill your new nodes
with data taken from dc1 : 
 
Run a rolling « nodetool rebuild dc1 »
on all nodes in dc2

 
 
Your new DC is now
fully ready to rock

 
How to remove DC2 from the cluster

 
Switch all trafﬁc to DC1

 
You may want to run repair

 
You may want to run repair
(hopefully you’ve seen my talk yesterday)

Operations - removing a DC 
 
WITH replication =
'dc1' : 3,
'dc2' : 3
};

 
Decommission all nodes from dc2
Run « nodetool decommission »
on all nodes in dc2

Hints & repair 
Anti-entropy repair
 
Merkle trees are requested
from all replicas in all DCs by default

Hints & repair
 
Speciﬁc switch to run repair
in the local DC:
nodetool repair -local

Hints & repair 
 
Should you run repair on all DCs ?

Hints & repair 
 
Yes, if :
SimpleStrategy KS
or
-local switch
or
KS not replicated to all DCs

Hints & repair 
Otherwise no

Hints & repair 
Try to avoid « over-repairing »
your cluster
Each token range needs a single pass…

Hints & repair 
 
Hints work between DCs
like they do between nodes
in a single DC

Hints & repair 
 
Hints can be disabled on speciﬁc DCs
in conf/cassandra.yaml :
hinted_handoff_disabled_datacenters:
- DC1
- DC2

Hints & repair 
 
This means DC1 and DC2
won’t receive hints
(use this wisely)

Hints & repair 
 
Advice for hints in multi DC clusters :
raise max_hints_delivery_threads to 4

Failover
 
Failover in single DC clusters
Use CL ONE or QUORUM

Failover
 
Failover in multi DC strongly consistent
clusters
Use CL ONE or QUORUM

Failover
 

Failover
 
Failover in multi DC eventually consistent
clusters
What if my local DC loses QUORUM ?

Failover
Quorum at RF 6 is 4

Failover
 
You have to build inter DC failover

Failover
 
 
detect failure
metrics, token range monitoring, …

Failover
 
switch trafﬁc
network, app or driver level

Failover
 
prevent premature back switch
inconsistencies

Failover
 
Build your own load balancing policy
on top of the DCAwareRoundRobinPolicy

Failover
 
If you have an analytical DC
and want synchronous operational DCs
 
Use racks

Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016 (20)

More from DataStax (20)

Recently uploaded (20)

Operations, Consistency, Failover for Multi-DC Clusters (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016