Cacheconcurrencyconsistency cassandra svcc

Cache & Concurrency & Consistency
considerations for Cassandra

SriSatish Ambati
Principal Engineer, DataStax (formerly, Riptano)
Apache Cassandra
OpenJDK/JVM
Twitter: @srisatish
srisatish.ambati@gmail.com

Trail ahead
Elements of Cache Performance
Metrics, Monitors
JVM goes to BigData Land!
Examples
Lucandra, Twissandra
Cassandra Performance with JVM
Commentary
Runtime Views
Non Blocking HashMap
Locking: concurrency
Garbage Collection

A feather in the CAP
● Eventual Consistency
– Levels
– Doesn’t mean data loss
(journaled)
● SEDA
– Partitioning, Cluster &
Failure detection,
Storage engine mod
– Event driven & non
blocking io
– Pure Java

Which is the most popular eventually
consistent system out there?

State machines as way to
visualize distributed systems
In a distributed system
● sometimes impossible to say that one of
two events occurred first.
● The relation “happened before” → is only a
partial ordering of event
For any events a, b and Clock C:
– If a → b then C(a) < C(b)
Leslie Lamport.

client side consistency
● A writes. B & C read after time t. Inconsistency Window
● Types of “eventual consistency”
● Causal Consistency
– A in a causal B sees consistent data
● ReadyourWrites
– A only sees updated valuse of data
● Session Consistency
– In the context of a session
● Monotonic read
– If a process sees a new value:everyone else
guaranteed to see update.
● Monotonic Write

Server side consistency:
R + W > N
● N the number of nodes that store replicas of the
data (RF)
● W the number of replicas that need to
acknowledge the receipt of the update before the
update completes
● R the number of replicas that are contacted
when a data object is accessed through a read
operation
T is the number of nodes.

Quorum
● R + W > N
– R=1, W=2, N=2 (primarybackup, rdbms)
● Faulttolerance
– R=2, W=2, N=3
● Read performance:
– R=1, Lots of nodes. Even 100s.
– R=1, N=W. High consistency
● Fast write:
– W=1, N=R

Count what is countable, measure what is
measurable, and what is not measurable,
make measurable

-Galileo

Elements of Cache Performance:
Metrics
● Operations:
– Ops/s: Puts/sec, Gets/sec, updates/sec
– Latencies, percentiles
– Indexing
● # of nodes – scale, elasticity
● Replication
– Synchronous, Asynchronous (fast writes)
● Tuneable Consistency
● Durability/Persistence
● Size & Number of Objects, Size of Cache
● # of user clients

Elements of Cache Performance:
“Think Locality”
● Hot or Not: The 80/20 rule.
– A small set of objects are very popular!
– What is the most RT tweet?
● Hit or Miss: Hit Ratio
– How effective is your cache?
– LRU, LFU, FIFO.. Expiration
● Longlived objects lead to better locality.
● Spikes happen
– Cascading events
– Cache Thrash: full table scans

Real World Performance
● Facebook Inbox
– Writes:0.12ms, Reads:15ms @ 50GB data
● Twitter performance
– Twissandra (simulation)
● Cassandra for Search & Portals
– Lucandra, solandra (simulation)
● ycbs/PNUTS benchmarks
– 5ms read/writes @ 5k ops/s (50/50 Update heavy)
– 8ms reads/5ms writes @ 5k ops/s (95/5 read heavy)
● Lab environment
– ~5k writes per sec per node, <5ms latencies
– ~10k reads per sec per node, <5ms latencies
● Performance has improved in newer versions

yahoo cloud store
benchmark
50/50 – Update Heavy

yahoo cloud store
benchmark
95/5 – read heavy

JVM in BigData Land!
Limits for scale
●
Locks : synchronized
– Can’t use all my multicores!
– java.util.collections also hold locks
– Use nonblocking collections!
● (de)Serialization is expensive
– Hampers object portability
– Use avro, thrift!
● Object overhead
– average enterprise collection has 3 elements!
– Use byte[ ], primitives where possible!
● Garbage Collection
– Can’t throw memory at the problem!
– Mitigate, Monitor, Measure foot print

Tools
● What is the JVM doing:
– dtrace, hprof, introscope, jconsole,
visualvm, yourkit, azul zvision
● Invasive JVM observation tools
– bci, jvmti, jvmdi/pi agents, jmx, logging
● What is the OS doing:
– dtrace, oprofile, vtune
● What is the network disk doing:
– Ganglia, iostat, lsof, netstat, nagios

furiously fast writes
n2 l y t o
app ry
client o
issues mem
n1
write find n o d e

commit log
partitioner
●
Append only writes
– Sequential disk access
●
No locks in critical path
●
Key based atomicity

furiously fast writes
● Use separate disks for commitlog
– Don’t forget to size them well
– Isolation difficult in the cloud..
● Memtable/SSTable sizes
– Delicately balanced with GC
● memtable_throughput_in_mb

Cassandra on EC2 cloud

*Corey Hulen, EC2

Compactions
K2 < Serialized data > K4 < Serialized data >
K1 < Serialized data >

D E L E T E D

Sorted Sorted
Sorted

MERGE  SORT

Index File
Loaded in memory K2 < Serialized data >
K1   Offset
K5  Offset Sorted
K30  Offset
Bloom Filter

Data File

Compactions
● Intense disk io & mem churn
● Triggers GC for tombstones
● Minor/Major Compactions
● Reduce priority for better reads

Relevant Parameters
Co m p a c t io n Ma n a g e r.
m in im u m Co m p a c t io n Th re s h o ld = xxxx

Example: compaction in
realworld, cloudkick

reads performance
● BloomFilter used to identify the right file
● Maintain column indices to look up columns
– Which can span different SSTables
● Less io than typical btree
● Cold read: Two seeks
– One for Key lookup, another row lookup
● Key Cache
– Optimized in latest cassandra
● Row Cache
– Improves read performance
– GC sensitive for large rows.
● Most (google) applications require single row
transactions*
*Sanjay G, BigTable Design, Google.

Client Performance
Marshal Arts:
Ser/Deserialization
● Clients dominated by Thrift, Avro
– Hector, Pelops
● Thrift: upgrade to latest: 0.5, 0.4
● No news: java.io.Serializable is S.L..O.…W
● Use “transient”
● avro, thrift, protobuf
● Common Patterns of Doom:
– Death by a million gets

Serialization +
●
Deserialization uBench
http://code.google.com/p/thriftprotobufcompare/wiki/BenchmarkingV2

Adding Nodes
●
New nodes
– Add themselves to busiest node

– And then Split its Range
●
Busy Node starts transmit to new node
●
Bootstrap logic initiated from any node, cli, web
●
Each node capable of ~40MB/s
– Multiple replicas to parallelize bootstrap
●
UDP for control messages
●
TCP for request routing

internode comm
● Gossip Protocol
– It’s exponential
– (epidemic algorithm)
● Failure Detector
– Accrual rate phi
● AntiEntropy
– Bringing replicas to uptodate

Bloom Filter: in full bloom
● “constant” time
● size:compact
● false positives
● Single lookup
for key in file
● Deletion
● Improve
– Counting BF

– Bloomier filters

Birthdays, Collisions &
Hashing functions
● Birthday Paradox
For the N=21 people in this room
Probability that at least 2 of them share same birthday is
~0.47
● Collisions are real!
● An unbalanced HashMap behaves like a list O(n) retrieval
● Chaining & Linear probing
● Performance Degrades
● with 80% table density

●

the devil’s in the details

CFS
● All in the
family!
● denormalize

Memtable
● Inmemory
● ColumnFamily specific
● throughput determines
size before flush
● Larger memtables can
improve reads

SSTable
● MemTable “flushes” to
a SSTable
● Immutable after
● Read: Multiple
SSTable lookups
possible
● Chief Execs:
– SSTableWriter
– SSTableReader

U U I D
● java.util.UUID is slow
– static use leads to contention
SecureRandom
● Uses /dev/urandom for seed initialization
-Djava.security.egd=file:/dev/urandom
● PRNG without file is atleast 20%40% better.
● Use TimeUUIDs where possible – much faster
● JUG – java.uuid.generator

● http://github.com/cowtowncoder/javauuidgenerator
● http://jug.safehaus.org/
● http://johannburkard.de/blog/programming/java/JavaUUIDgeneratorscompared.html

synchronized
● Coarse grained locks
● io under lock
● Stop signal on a highway
● java.util.concurrent does not mean no
locks
● Non Blocking, Lock free, Wait free
collections

Scalable LockFree Coding Style
● Big Array to hold Data
● Concurrent writes via: CAS & Finite State
Machine
– No locks, no volatile
– Much faster than locking under heavy load
– Directly reach main data array in 1 step
● Resize as needed
– Copy Array to a larger Array on demand
– Use State Machine to help copy
– “ Mark” old Array words to avoid missing late updates

NonBlocking HashMap
Azul Vega2 – 768 cpus

1K Table 1M Table
1200 1200

1000
NB99 1000

800 800

Mops/sec
CHM99
Mops/sec

600 600

400 400 NB
NB75
200 200
CHM75
CHM
0 0
0 100 200 300 400 500 600 700 800 0 100 200 300 400 500 600 700 800
Threads Threads

Cassandra uses High Scale
NonBlocking Hashmap
public class BinaryMemtable implements IFlushable
{
…
private final Map<DecoratedKey,byte[]> columnFamilies =
new NonBlockingHashMap<DecoratedKey, byte[]>();
/* Lock and Condition for notifying new clients about Memtable
switches */
private final Lock lock = new ReentrantLock(); Condition condition;
…
}
public class Table
{
…
private static final Map<String, Table> instances = new
NonBlockingHashMap<String, Table>();
…
}

GCsensitive elements
within Cassandra
● Compaction triggers System.gc()
– Tombstones from files
● “GCInspector”
● Memtable Threshold, sizes
● SSTable sizes
● Low overhead collection choices

Garbage Collection
● Pause Times
if stop_the_word_FullGC > ttl_of_node
=> failed requests; failure accrual & node repair.
● Allocation Rate
– New object creation, insertion rate
● Live Objects (residency)
– if residency in heap > 50%
– GC overheads dominate.
● Overhead
– space, cpu cycles spent GC
● 64bit not addressing pause times
– Bigger is not better!

Memory Fragmentation
● Fragmentation
– Performance degrades over time
– Inducing “Full GC” makes problem go away
– Free memory that cannot be used
● Reduce occurrence
– Use a compacting collector
– Promote less often
– Use uniform sized objects
● Solution – unsolved
– Use latest CMS with CR:6631166
– Azul’s Zing JVM & Pauseless GC

Best Practices:
Garbage Collection
● GC Logs are cheap even in production
   Xloggc:/var/log/cassandra/gc.log
    XX:+PrintGCDetails
    XX:+PrintGCTimeStamps XX:+PrintTenuringDistribution
   XX:+PrintHeapAtGC
● Slightly expensive ones:
  XX:PrintFLSStatistics=2 XX:CMSStatistics=1
   XX:CMSInitiationStatistics

Sizing: Young
Generation
● Should we set –Xms == Xmx ?
● Use –Xmn (fixed eden)

allocations {new Object();}
survivor ratio

eden survivor spaces
Tenuring
promotion
Threshold

allocation by jvm
old generation

Tuning CMS
● Don’t promote too often!
– Frequent promotion causes fragmentation
● Size the generations
– Min GC times are a function of Live Set
– Old Gen should host steady state comfortably
● Parallelize on multicores:
– XX:ParallelCMSThreads=4
– XX:ParallelGCThreads=4
● Avoid CMS Initiating heuristic
– XX:+UseCMSInitiationOccupanyOnly
● Use Concurrent for System.gc()
– XX:+ExplicitGCInvokesConcurrent

Summary
Design & Implementation of Cassandra takes advantages
of strengths while avoiding common JVM issues.
● Locks:
– Avoids locks in critical path
– Uses nonblocking collections, TimeUUIDs!
– Still Can’t use all my multicores..?
>> Other bottlenecks to find!
● De/Serialization:
– Uses avro, thrift!
● Object overhead
– Uses mostly byte[ ], primitives where possible!
● Garbage Collection
– Mitigate: Monitor, Measure foot print.
– Work in progress by all jvm vendors!
Cassandra starts from a great footing from a JVM standpoint
and will reap the benefits of the platform!

References Q&A
● Leslie Lamport, “Time, Clocks and the Ordering of Events in a Distributed
System”
● Verner Wogels, Eventually Consistent
http://www.allthingsdistributed.com/2008/12/eventually_consistent.html
● Bloom, Burton H. (1970), "Space/time tradeoffs in hash coding with
allowable errors"
● Avinash Lakshman, http://static.last.fm/johan/nosql
20090611/cassandra_nosql.pdf
● Eric Brewer, CAP
http://www.cs.berkeley.edu/~brewer/cs262b2004/PODCkeynote.pdf
● Tony Printzeis, Charlie Hunt, Javaone Talk
http://www.scribd.com/doc/36090475/GCTuningintheJava
● http://github.com/digitalreasoning/PyStratus/wiki/Documentation
● http://www.cs.cornell.edu/home/rvr/papers/flowgossip.pdf
● Cassandra on Cloud, http://www.coreyhulen.org/?p=326
● Cliff Click’s, Nonblocking HashMap http://sourceforge.net/projects/highscale
lib/
● Brian F. Cooper., Yahoo Cloud Storage Benchmark,
http://www.brianfrankcooper.net/pubs/ycsbv4.pdf

DataModel:
Know your use patterns

Alternative Twitter DataModel:

<Keyspace Name="Multiblog">
<ColumnFamily CompareWith="TimeUUIDType" Name="Blogs" />
<ColumnFamily CompareWith="TimeUUIDType" Name="Comments"/>
</Keyspace>

Cacheconcurrencyconsistency cassandra svcc

Recommended

More Related Content

What's hot (20)

Similar to Cacheconcurrencyconsistency cassandra svcc (20)

More from srisatish ambati (16)

Recently uploaded (20)

Cacheconcurrencyconsistency cassandra svcc