Why does my choice of storage matter with cassandra?

Why does my choice of storage matter with
Cassandra?
Johnny Miller, Solutions Architect
@CyanMiller
www.linkedin.com/in/johnnymiller

Quote
“The single biggest predictor of success or failure
with a Cassandra deployment is in storage choice”
Patrick McFadin, Chief Evangelist for Cassandra, @PatrickMcFadin
©2014 DataStax Confidential. Do not distribute without consent. 2

Cassandra Storage Engine

Inserts/Updates
Memtables are organized in sorted
order by row key and flushed to
SSTables sequentially (Read/Write)
Ordered Map of KVP,
(Immutable, Read Only)
Append only file structure,
providing interim durability for
writes before they get flushed to
SSTables (Write Only)

Reads

Deletes
•  Unlike most DBs, deleted data is not immediately
removed from disk.
•  A marker called a tombstone is written to indicate the the
column is deleted
•  A tombstones exist for a configurable period of time, and
are only deleted from disk via compaction after that time
has expired.
•  Deletes are just as fast as inserts J

Compaction
•  Regular compaction of data in Cassandra is essential for a healthy and performant
cluster.
•  SSTables are immutable
•  Get rid or duplicate/overwritten data
•  Drop deleted data and tomnstones
•  Data in SSTables is sorted by partition key, the effect of which is that while the
SSTables are being consolidated, the disk I/O is not random.

Compaction Strategies
•  There is a choice of three strategies Cassandra can use for compaction
and all have different disk I/O profiles and capacity requirements.
•  SizeTieredCompactionStrategy (default)
•  Using this strategy causes bursts in I/O activity while a compaction is in process
•  These I/O bursts can negatively affect read-heavy workloads, but typically do not impact
write performance.
•  Data highly likely to be spread across multiple SSTables i.e. multiple disk seeks
•  LeveledCompactionStrategy
•  ~90% of the time, data will be in only a single SSTable i.e. minimal disk seeks
•  However, there is significantly higher Disk I/O than size tiered compaction in order to
guarantee how many SSTables data may be spread across
•  Due to high disk I/O rarely appropriate for on traditional HDD
•  DateTieredCompactionStrategy (C* 2.0.11+ and 2.1.1+)
•  Stores data written within a certain period of time in the same SSTable.
•  Can store data that has been set to expire using TTL in an SSTable with other data
scheduled to expire at approximately – can just drop the SSTable without any
compaction!

Choice of storage matters
•  Most databases rewrite modified data in place and writes are
buffered and then flushed to disk as random writes.
•  With Cassandra:
•  Disk writes are typically sequential append only
operations
•  On-disk tables are written in a sorted order so compaction
running time increases linearly with the amount of data
•  So choice of storage is pretty important!!

Disks and Configuration Options

Quote
“For many applications, we are no longer constrained by hard drive
capacity, but by seek speeds.
Essentially, a 7200 RPM hard drive is capable of delivering approximately
100 seeks per second, and this has not changed in more than 10 years,
even as disk capacity has been doubling every 18–24 months.
In fact, if you have a big data application which required a half a petabyte of
storage, what had previously required 1024 disk drives when we were using
500 GB drives, now that 3 TB disks are available, only 171 disk drives are
needed.
So a storage array capable of storing half a petabyte is now capable of 80%
fewer seeks.”
- Ted Ts’o Maintainer of the ext4 file system in the Linux kernel

Hard Drive/Spinning Disk
This part actually has to move!This bit spins around very fast

So what can we do?
•  Memory?
•  Caching can help, but the hit rate has to be extremely high to mitigate the mechanical
latency of spinning disks
•  Get rid of the moving parts!
•  Mechanical media will never be able to keep up under load
•  Today’s databases service multiple users with difference access patterns
•  A relatively small number of concurrent disk reads can result in seconds of latency
•  SSDs don’t have moving parts
•  SSDs can eliminate entire classes of problems
•  With Cassandra in particular, you will save a lot of money on staff resources by investing
in SSDs up front
•  Compactions can be tough on flash, but it’s not as bad as you think

The best way to do it – SSDs!
•  What is an SSD?
•  Solid State Drive
•  Bits stored in NAND Flash Memory
•  No moving parts
•  “Seeks” 2-3 orders of magnitude faster than spinning disk
•  What’s the catch?
•  Smaller capacity
•  More expensive
•  Flash wears out
In practice, this is not a problem – if it makes you nervous, keep spares

The IO Scheduler
•  NOOP - use this scheduler if you know another IO device (like a RAID
card) will be doing its own IO scheduling. The NOOP scheduler is just
a pass-through.
•  Deadline - otherwise, use the deadline scheduler
•  Tell the OS the drive is non-rotational
•  Tune read-ahead way down – start at 0 and work your way up
Don’t forget to tune the OS for SSDs
echo deadline > /sys/block/<drive>/queue/scheduler
echo 0 > /sys/block/sda/queue/rotational
blockdev –setra 0 /dev/<drive>

Use SSDs
•  More flexibility and substantial performance benefit
•  Typically 10x the performance for less than 2x the cost (potentially lower) when compared
with HDDs.
•  You can use LeveledCompactionStrategy
•  SSD drives can scale up to cope with larger compaction overheads while simultaneously
serving many random reads.
•  Netflix found that they could half the total system cost to achieve the same level of
throughput.
•  Additionally the mean read request latency was reduced from 10ms to 2.2ms and 99th
percentile request latency was reduced from 65ms to 10ms.
•  http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html.

The worst way to do it
•  Shared Storage
•  Cassandra is a shared-nothing architecture with no single point of failure
•  Adding shared storage adds a single point of failure
•  Irrespective of the terrible performance – this alone is enough reason not to do
it.
Shared
Storage

Local storage configuration
•  RAID or JBOD?
•  RAID – Redundant Array of (Independepent/Inexpensive) Devices
•  JBOD – Just a Bunch Of Disks
•  RAID
•  Common Cassandra RAID levels are 0, 1, 10
•  RAID-0 is most common, but means all data on node must be rebuilt
from other nodes when a drive fails
•  JBOD
•  Drives are listed individually in cassandra.yaml
•  Failed drives can be replaces individually

How to choose between RAID or JBOD?
•  Performance
•  For SSDs, not so much…
•  Compactions are usually throttled significantly below bus speed
•  So a single SSD usually has sufficient throughput
•  Throughput is really the only advantage RAID buys
•  Manageability
•  Pick the option that best fits the deployment scenario
•  If using SSD, and drives can be replaced, choose JBOD
•  Otherwise, RAID is probably the right choice.

How to choose between RAID or JBOD?
•  Cloud provider
•  EC2 ephemeral SSD can’t be replaced, use RAID (and dont use EBS)
•  GCE persistent SSD volumes can be replaced, JBOD is useful
•  Not all drives are hot swapable
•  PCIe devices can’t conveniently be replaced
•  SSD Spares for JBOD mode
•  Keep spare SSDs online, but not in use
•  Allows the node to be easily brought back online
with a quick config change

Comparison Data

FusionIO ioDrive II
Reads
Writes
Latency (microseconds)

PNY XLR8 SSD (consumer grade MLC)

Samsung 840 Pro SSD (consumer grade MLC)

7200RPM SATA

7200RPM SAS

10K SATA

15K SAS

All Drives

All SSDs

Conclusion
•  Using SSDs is a good idea
•  Better response times
•  Less variance in performance
•  Significantly higher throughput so fewer servers needed
VS

Thank You
We power the big data apps
that transform business.

Why does my choice of storage matter with cassandra?

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Why does my choice of storage matter with cassandra? (20)

More from Johnny Miller (6)

Recently uploaded (20)

Why does my choice of storage matter with cassandra?