Stampede con 2014 cassandra in the real world

STAMPEDECON 2014

CASSANDRA

IN THE REAL WORLD

Nate McCall

@zznate

!
Co-Founder & Sr.Technical Consultant

!
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License

AboutThe Last Pickle.

!
Work with clients to deliver and improve
Apache Cassandra based solutions.

!
Based in New Zealand & USA.

“…in the Real World?”

!
Lots of hype,

stats get attention,

as do big names

“Real World?”
!
“…1.1 million client writes per second.
Data was automatically replicated across all
three zones making a total of 3.3 million
writes per second across the cluster.”
http://techblog.netﬂix.com/2011/11/benchmarking-cassandra-scalability-on.html

“Real World?”
!
“+10 clusters, +100s nodes,

250TB provisioned,

9 billion writes/day, 5 billion reads/day”
http://www.slideshare.net/jaykumarpatel/cassandra-at-ebay-cassandra-summit-2013

“Real World?”
!
…

• “but I don’t have an∞ AMZN budget”

• “maybe one day I’ll have that much data”

“Real World!”
!
Most folks needed:

real fault tolerance,

scale out characteristics

“Real World!”
!
Most folks have:

3 to 12 nodes with 2-15TB,

commodity hardware,

small teams

!
Cassandra at 10k feet
Case Studies

Common Best Practices
Cassandra in the Real World.

Cassandra Architecture (brieﬂy).
API's
Cluster Aware
Cluster Unaware
Clients
Disk

Cassandra Cluster Architecture (brieﬂy).
API's
Cluster Aware
Cluster Unaware
Clients
Disk
API's
Cluster Aware
Cluster Unaware
Disk
Node 1 Node 2

Dynamo Cluster Architecture (brieﬂy).
API's
Dynamo
Database
Clients
Disk
API's
Dynamo
Database
Disk
Node 1 Node 2


!
API
Dynamo

Database

APITransports.
!

Thrift
Native Binary

Thrift transport.
!

Extremely performant for
speciﬁc workloads
Astyanax,

disruptor-based HSHA in 2.0

Native BinaryTransport.
!

Focus of future development
Uses Netty,
CQL 3 only,

asynchronous

API Services.
!

JMX
Thrift
CQL 3
!


!
API
Dynamo

Database
Please see:

http://www.slideshare.net/aaronmorton/cassandra-community-webinar-introduction-to-apache-cassandra-12-20353118

http://www.slideshare.net/planetcassandra/c-summit-eu-2013-cassandra-internals

http://www.slideshare.net/aaronmorton/cassandra-community-webinar-august-29th-2013-in-case-of-emergency-break-glass

Cassandra in the Real World.

!
Cassandra at 10k feet
Case Studies

Common Best Practices

Case Studies.
Ad Tech

Sensor Data

Mobile Device Diagnostics

AdTech.

Large “Hot Data” set

active users,

targeting,

display count

AdTech.

Huge Long Tail

who saw what,

used for billing,

campaign effectiveness over time,

all sorts of analytics

AdTech: Software.

Java

CQL via DataStax Java Driver

Python

Pycassa (Thrift)

AdTech: Cluster.
Cluster

12 nodes,

2 datacenters,

{DC1:R1:3,DC2:R2:3}

AdTech: Systems.
Physical Hardware

commodity 1U 8xSSD,

36GB RAM,

10gigE + 4x1gigE

Case Studies.
AdTech

Sensor Data

Mobile Device Diagnostics

Sensor Data.

High Write Throughput:

consistent “shape”,

immutable data,

large sequential reads,

high uptime (for writes)

Sensor Data: Software.

REST application:

separate reader service,

writes to kafka,

ELB to multiple regions

Sensor Data: Software.

Java:

Thrift via Astyanax,

read from kafka and batch
insertions to optimal size

Sensor Data: Cluster.

Cluster

9 nodes,

1 availability zone,

{RF:3}

Sensor Data: Systems.

m1.xlarge:

15GB, 2TB RAID0

“high”,

tablesnap for backup

Device Diagnostics.

Latency = battery

Device Diagnostics.

Write Bursts

large single payloads,

large hot data set

Device Diagnostics.

Huge long tail

but irrelevant after 2 months,

external partner API*

!
*thar be dragons

Device Diagnostics: Software.

Java

CQL / DataStax Java Driver

Device Diagnostics: Software.

REST application

Payloads to S3,

pointer in kafka to payload

Device Diagnostics: Cluster.

Cluster

12 nodes,

3 availability zones

{us-east-1:1}

Device Diagnostics: Systems.

i2.2xlarge

61gb, 1.8TB RAID0 SSD

“Enhanced Networking”,

dedicated ENI


No Backups.

!
!


No Backups.

!
“Replay the front end.”

Common Best Practices.
API's
Cluster Aware
Cluster Unaware
Clients
Disk

Client Best Practices.

Decouple!

buffer writes for

event based systems,

use asynchronous operations


Use Ofﬁcial Drivers

(but there are exceptions)


CQL3:

collections,

user deﬁned types,

tooling available

API Best Practices.

Understand Replication!

API Best Practices.

Monitor & Instrument

Cluster Best Practices.

Understand Replication!

learn all you can about
topology options

Cluster Best Practices.

Verify Assumptions:

test failure scenarios explicitly

Systems Best Practices.

Better to have a lot of a little

commodity hardware*,

32-64gb or RAM (or more)
*10gigE is now commodity


BUT: do you have staff that
can tune kernels?

larger hardware needs tuning:

“receive packet steering”


EC2

SSD instances if you can,

UseVPCs, Deployment groups
and ENIs

Storage Best Practices.

Dependent on workload

can mix and match:

rotational for commitlog and
system


You can mix and match:

rotational for commitlog and
system,

SSD for data


SSD

consider JBOD,

consumer grade works ﬁne


“What about SANs?”


“What about SANs?”

!
NO.

!
(You would be moving a distributed system

onto a centralized component)


Backups:

tablesnap on EC2,

rsync (immutable data FTW!)


Backups:

combine rebuild+replay for
best results

(Bonus: loading production data to staging is
testing your backups!)

Nate McCall

@zznate

!
Co-Founder & Sr.Technical Consultant

www.thelastpickle.com

Stampede con 2014 cassandra in the real world

More Related Content

What's hot (18)

Viewers also liked (14)

Similar to Stampede con 2014 cassandra in the real world (20)

More from zznate (9)

Recently uploaded (20)

Stampede con 2014 cassandra in the real world