Spotify: Automating Cassandra repairs
6 likes3,718 views
Anti-entropy repairs are known to be a very peculiar maintenance operation of Cassandra clusters. They are problematic mostly because of the potential of having negative impact on the cluster's performance. Another problematic aspect is the difficulty of managing the repairs of Cassandra clusters in a careful way that would prevent the negative performance impact. Based on the long-term pain we have been experiencing with managing repairs of nearly 100 Cassandra clusters, and being unable to find a solution that would meet our needs, we went ahead and developed an open-source tool, named Cassandra Reaper [1], for easy management of Cassandra repairs. Cassandra Reaper is a tool that automates the management of anti-entropy repairs of Cassandra clusters in a rather smart, efficient and careful manner while requiring minimal Cassandra expertise. I will have to cover some basics of eventual consistency mechanisms of Cassandra, after which I will be able to focus on the features of Cassandra Reaper and our six months of experience having the tool managing the repairs of our production clusters.
![[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/enbuildingmoderndatapipelinewithsnowflakedbtairflow-231024053710-6b96c278-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (7)
Ad
Similar to Spotify: Automating Cassandra repairs (20)
Ad
More from DataStax Academy (20)
Recently uploaded (20)
Spotify: Automating Cassandra repairs
- 2. About zvo
- 4. About zvo Likes pancakes Does this for the 3rd time
- 5. About zvo Likes pancakes Does this for the 3rd time Works at Spotify
- 6. Working at Spotify Is autonomous Squads responsible for their full stack Including Cassandra
- 7. Cassandra
- 10. Running Cassandra Requires many things One of them is keeping data consistent Otherwise it can get lost or reappear
- 11. Running Cassandra Requires many things One of them is keeping data consistent Eventually
- 17. Coordinated process Four steps: Anti-entropy Repair
- 18. Coordinated process Four steps: 1: Hash Anti-entropy Repair # # #
- 19. Coordinated process Four steps: 1: Hash 2: Compare Anti-entropy Repair ###
- 20. Coordinated process Four steps: 1: Hash 2: Compare 3: Stream Anti-entropy Repair
- 21. Coordinated process Four steps: 1: Hash 2: Compare 3: Stream 4: Merge Anti-entropy Repair
- 22. Coordinated process Four steps: 1: Hash 2: Compare 3: Stream 4: Merge Can go wild... Anti-entropy Repair
- 23. Repair gone wild
- 24. Repair gone wild Eats a lot of disk IO ● because of hashing all the data
- 25. Repair gone wild Eats a lot of disk IO Saturates the network ● because of streaming a lot of data around
- 26. Repair gone wild Eats a lot of disk IO Saturates the network Fills up the disk ● because of receiving all replicas, possibly from all other data centers
- 27. Repair gone wild Eats a lot of disk IO Saturates the network Fills up the disk Causes a ton of compactions ● because of having to merge the received data
- 28. Repair gone wild Eats a lot of disk IO Saturates the network Fills up the disk Causes a ton of compactions … one better be careful
- 29. Careful repair
- 30. nodetool repair Careful repair All three intervals
- 31. Partitioner range ● nodetool repair -pr Careful repair This interval only
- 32. Start & end tokens ● nodetool repair -pr -st -et Careful repair A part of interval only
- 33. Requires splitting the ring into smaller intervals Smaller intervals mean less data Less data means fewer repairs gone wild Careful repair
- 34. Smaller intervals also mean more intervals More intervals mean more actual repairs Repairs need to be babysat :( Careful repair
- 35. The Spotify way Feature teams meant to do features Not waste time operating their C* clusters Cron-ing nodetool repair is no good ● mostly due to no feedback loop This all led to creation of the Reaper
- 36. The Reaper REST(ish) service Does a lot of JMX Orchestrates repairs for you
- 37. The reaping You: curl http://reaper/cluster --data ‘{“seedHost” : “my.cassandra.host.net”}’ The Reaper: ● Figures out cluster info (e.g. name, partitioner)
- 38. The reaping You: curl http://reaper/repair_run --data ‘{“clusterName”: “myCluster”}’ The Reaper: ● Prepares repair intervals
- 39. The reaping You: curl -X PUT http://reaper/repair_run/42 -d state=RUNNING The Reaper: ● Starts triggering repairs of repair intervals
- 41. Reaper’s features Carefulness - doesn’t kill a node ● checks for node load ● backs off after repairing an interval
- 42. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break ● because things break all the time
- 43. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes ● multiple small intervals in parallel
- 44. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes Scheduling - setup things only once ● regular full-ring repairs
- 45. Reaper’s features Carefulness - doesn’t kill a node Resilience - retries when things break Parallelism - no idle nodes Scheduling - setup things only once Persistency - state saved somewhere ● a bit of extra resilience
- 46. What we reaped First repair done 2015-01-28 1,700 repairs since then, recently 90 per week 176,000 (16%) segments failed at least once 60 repair failures
- 47. What we reaped
- 48. What we reaped
- 49. Reaper’s Future CASSANDRA-10070 Whatever is needed until then
- 50. Greatest benefit Cassandra Reaper automates a very tedious maintenance operation of Cassandra clusters in a rather smart, efficient and careful manner while requiring minimal Cassandra expertise github.com/spotify/cassandra-reaper #CassandraSummit