Embracing collaborative chaos (April 2020) by Lyndsay Prewer

Editor's Notes

• #2: Hello, my name is Lyndsay Prewer. Over the last couple of years, I’ve been leading a group of teams that develop and operate a Platform-as-a-Service for a very large public sector client. In this talk I’ll describe how we’ve used Chaos Days to improve the resilience of our platform, and the effectiveness of our platform and it’s teams to gracefully handle catastrophic failures.
• #4: Chaos engineering is particularly relevant to distributed systems, as these have a scale and high level of complexity that make it impossible to determine their emergent properties and behaviour, let alone every possible failure mode, it’s impact and possible mitigation. Although distributed systems have been around for decades, recent advances in technology, such as serverless, combined with agile and lean practices have led to teams being able to get more complex stuff into production faster and at lower cost. We can build really cool applications like Nest XYZ, so we can do ABC. What could possibly go wrong!?
• #5: Chaos engineering is particularly relevant to distributed systems, as these have a scale and high level of complexity that make it impossible to determine their emergent properties and behaviour, let alone every possible failure mode, it’s impact and possible mitigation. Although distributed systems have been around for decades, recent advances in technology, such as serverless, combined with agile and lean practices have led to teams being able to get more complex stuff into production faster and at lower cost. We can build really cool applications like Nest XYZ, so we can do ABC. What could possibly go wrong!?
• #6: We can build really cool applications like Nest XYZ, so we can do ABC. What could possibly go wrong!? Complex/distributed systems will fail - not if but when - our systems operate on the edge of chaos
• #7: Consider your own system...
• #8: As component parts and connections increase we get an exponential increase in the complexity of the emergent behaviour and thus the number of possible failure modes. This equates to a decrease in our ability to predict failures and their impact zone. Building resilience in, similar to Build quality in Production thinking “It’s a mindset, not a toolset: you don’t need to be running EKS on AWS to benefit from ….”
• #9: It doesn’t mean we build systems that never fail, that are perfect and indestructible. It means we build systems that cope with failure well, that recover well, that are elastic.
• #11: Chaos Days (focus on what, not why, as why comes later) Chaos testing (focus is very narrow/local to new/changed components)
• #12: Chaos Monkey, Symian army et al AWS and GCP alternatives (spot instances, etc.) (Semi-automated) - Super K8S Chaos Bro
• #13: Making this part of normal flow - link back to Production thinking / Building resilience in
• #14: Reference https://medium.com/@NetflixTechBlog/introducing-dispatch-da4b8a2a8072
• #15: It’s not just about more resilient components. It starts with people, their knowledge, their expertise, their behaviours. It covers process - how we respond to and manage incidents, how we learn from them, how we fold these learnings into our engineering practices. On the product front, it’s more than just resilience improvements. It’s also making systems easier to observe, easier to understand and reason about. Systems that automatically heal and tolerate failure is the goal, but improvements in things such as telemetry, alerting and runbooks.
• #17: Describe size, scale and architecture of Public sector client At various other clients, ranging from retail to payment systems, we’ve setup and run kube-monkey in all environments, opted for preemptible VMs, and run Game Days to help teams learn how to diagnose and debug Production issues.
• #22: For large platforms, owning teams should provide Chaos Agent to plot and scheme in secret with others. Who knows your system the best? Who do you turn to when the shit hits the fan? Should be high bus factor person.
• #23: Map out your architecture and dependencies Define steady state What’s normal load/throughput? How do you know the system is healthy? (heart rate, VO2-Max, metrics, 5XX / 499 (check this) responses, alerts) What do you have control over? What services / teams do you want to protect?
• #24: Apollo 13 picture Map out your architecture and dependencies Doesn’t need to be a big diagram - just get the experts together and brainstorm. Give them a clear intent, a goal, a direction and some constraints, then leave them to figure it out.
• #25: Define hypothesis for specific interventions and expected response, e.g. Instance failures, app failures, AZ failures, volumes filling up, connections failing/slowing, database failing. Security attacks (break-the-bank approaches, malicious engineer) Map out sequencing, e.g. what should go together, what kept apart, what can be done independently. How will normal service be resumed?
• #26: Chaos Days are a perfect time to also run security attacks (break-the-bank approaches, malicious engineer)
• #27: Production or not? If not how production like are things (cookie cutter environments, telemetry) How will load be generated? Who will be impacted if chaos does reign? What comm’s channel is normally used?
• #29: Some warning? Anything else happening at that time (e.g. peak loads, major releases) How will you ensure normal service is resumed - story from our first day
• #30: [Photo from 1st chaos day?] Co-locate agents of chaos, plus comm’s channel Collaborate and coordinate in response to chaos and how it’s handled. Timebox to ensure enough chaos variants covered and normal service is resumed [Slack and trello screen shots?] Record what you’re doing (slack, trello - hypothesis, expected response, actual)
• #31: Just an ordinary day (i.e. all teams working as normal) Record what you’re doing (slack) Treat chaos environment as production
• #33: Team based retro’s then team of teams Separate resilience improvements (e.g. tech, process, people) from chaos day improvements
• #34: [Slide, check our own list] Lessons learnt What’s not worked well Things we’d do next time
• #35: What’s your next step?
• #36: Describe various possible contexts, and possible next steps for each