Chaos Engineering Without Observability ... Is Just Chaos
- 1. Why chaos must be practiced ^^ not my talk title!
- 2. Closing the Loop on Chaos with Observability ^^ actual title
- 3. "Chaos" is a fancy marketing term for running tests later in the software development lifecycle. This is fine! But we used to call it something else:
- 5. I blame this guy: Testing in production has gotten a bad rap.
- 7. how they think we are how we really are
- 8. Our idea of what the software development lifecycle even looks like is overdue for an upgrade in the era of distributed systems.
- 9. Deploying code is not a binary switch. Deploying code is a process of increasing your confidence in your code.
- 13. why now?
- 14. “Complexity is increasing” - Science
- 15. Architectural complexity Parse, 2015LAMP stack, 2005
- 16. monitoring => observability known unknowns => unknown unknowns LAMP stack => distributed systems
- 17. We are all distributed systems engineers now the unknowns outstrip the knowns why does this matter more and more?
- 18. Distributed systems are particularly hostile to being cloned or imitated (or monitored). (clients, concurrency, chaotic traffic patterns, edge cases …)
- 19. Distributed systems have an infinitely long list of almost-impossible failure scenarios that make staging environments particularly worthless. this is a black hole for engineering time
- 20. Operational literacy Is not a nice-to-have
- 21. Without observability, you don't have "chaos engineering". You just have chaos. So what is observability?
- 22. Observability is NOT the same as monitoring.
- 23. @grepory, Monitorama 2016 “Monitoring is dead.” “Monitoring systems have not changed significantly in 20 years and has fallen behind the way we build software. Our software is now large distributed systems made up of many non-uniform interacting components while the core functionality of monitoring systems has stagnated.”
- 24. @grepory, 2016 This is a outdated model for complex systems.
- 25. Observability “In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. The observability and controllability of a system are mathematical duals." — wikipedia … translate??!?
- 26. Observability for software engineers: Can you understand what’s happening inside your systems, just by asking questions from the outside? Can you debug your code and its behavior using its output? Can you answer new questions without shipping new code?
- 27. You have an observable system when your team can quickly and reliably track down any new problem with no prior knowledge. For software engineers, this means being able to reason about your code, identify and fix bugs, and understand user experiences and behaviors ... via your instrumentation.
- 28. Monitoring Represents the world from the perspective of a third party, and describes the health of the system and/or its components in aggregate. Observability Describes the world from the first-person perspective of the software, executing each request. Software explaining itself from the inside out.
- 29. We don’t *know* what the questions are, all we have are unreliable symptoms or reports. Complexity is exploding everywhere, but our tools are designed for a predictable world. As soon as we know the question, we usually know the answer too.
- 30. Welcome to distributed systems. it’s probably fine. (it might be fine?)
- 31. Many catastrophic states exist at any given time. Your system is never entirely ‘up’
- 32. Distributed systems have an infinitely long list of almost-impossible failure scenarios that make staging environments particularly worthless. this is a black hole for engineering time
- 33. You do it. You have to do it. Do it well.
- 34. Let’s try some examples! Can you quickly and reliably track down problems like these?
- 35. The app tier capacity is exceeded. Maybe we rolled out a build with a perf regression, or maybe some app instances are down. DB queries are slower than normal. Maybe we deployed a bad new query, or there is lock contention. Errors or latency are high. We will look at several dashboards that reflect common root causes, and one of them will show us why. “Photos are loading slowly for some people. Why?” Monitoring (LAMP stack) monitor these things !
- 36. “Photos are loading slowly for some people. Why?” (microservices) Any microservices running on c2.4xlarge instances and PIOPS storage in us-east-1b has a 1/20 chance of running on degraded hardware, and will take 20x longer to complete for requests that hit the disk with a blocking call. This disproportionately impacts people looking at older archives due to our fanout model. Canadian users who are using the French language pack on the iPad running iOS 9, are hitting a firmware condition which makes it fail saving to local cache … which is why it FEELS like photos are loading slowly Our newest SDK makes db queries sequentially if the developer has enabled an optional feature flag. Working as intended; the reporters all had debug mode enabled. But flag should be renamed for clarity sake. wtf do i ‘monitor’ for?! Monitoring?!?
- 37. Problems Symptoms "I have twenty microservices and a sharded db and three other data stores across three regions, and everything seems to be getting a little bit slower over the past two weeks but nothing has changed that we know of, and oddly, latency is usually back to the historical norm on Tuesdays. “All twenty app micro services have 10% of available nodes enter a simultaneous crash loop cycle, about five times a day, at unpredictable intervals. They have nothing in common afaik and it doesn’t seem to impact the stateful services. It clears up before we can debug it, every time.” “Our users can compose their own queries that we execute server-side, and we don’t surface it to them when they are accidentally doing full table scans or even multiple full table scans, so they blame us.” Observability (microservices)
- 38. Still More Symptoms “Several users in Romania and Eastern Europe are complaining that all push notifications have been down for them … for days.” “Disney is complaining that once in a while, but not always, they don’t see the photo they expected to see — they see someone else’s photo! When they refresh, it’s fixed. Actually, we’ve had a few other people report this too, we just didn’t believe them.” “Sometimes a bot takes off, or an app is featured on the iTunes store, and it takes us a long long time to track down which app or user is generating disproportionate pressure on shared components of our system (esp databases). It’s different every time.” Observability “We run a platform, and it’s hard to programmatically distinguish between problems that users are inflicting themselves and problems in our own code, since they all manifest as the same errors or timeouts." (microservices)
- 39. These are all unknown-unknowns that may have never happened before, or ever happen again (They are also the overwhelming majority of what you have to care about for the rest of your life.)
- 40. Commandments well-instrumented high cardinality high dimensionality event-oriented perspective structured data software ownership sampled tested in prod.
- 41. Instrumentation Get inside the software’s head Explain it back to yourself/a naive user The right level of abstraction is key Wrap all network calls, etc Open up all black boxes to inspection
- 42. Events, not metrics Can you explain why literally any request has failed? Cardinality Context Structured data Tracing+events First-person perspective
- 43. UUIDs db raw queries normalized queries comments firstname, lastname PID/PPID app ID device ID HTTP header type build ID IP:port shopping cart ID userid ... etc Some of these … might be … useful … YA THINK??! High cardinality will save your ass. High cardinality
- 44. You must be able to break down by 1/millions and THEN by anything/everything else High cardinality is not a nice-to-have ‘Platform problems’ are now everybody’s problems
- 45. Jumps to an answer, instead of starting with a question You don’t know what you don’t know. Very limited utility. Dashboards Artifacts of past failures. Real time. Exploratory.
- 46. Aggregation is a one-way trip Destroying raw events eliminates your ability to ask new questions. Forever. Aggregates are the kiss of death
- 47. Raw Requests You can’t hunt needles if your tools don’t handle extreme outliers, aggregation by arbitrary values in a high-cardinality dimension, super-wide rich context… Black swans are the norm you must care about max/min, 99%, 99.9th, 99.99th, 99.999th …
- 48. Software engineers spend too much time looking at code in elaborately falsified environments, and not enough time observing it in the real world. Tighten feedback loops. Give developers the observability tooling they need to become fluent in production and to debug their own systems. We aren’t “writing code”. We are “building systems”.
- 49. Watch it run in production. Accept no substitute. Get used to observing your systems when they AREN’T on fire. Engineers
- 50. Real data Real users Real traffic Real scale Real concurrency Real network Real deploys Real unpredictabilities.
- 52. well-instrumented high cardinality high dimensionality event-driven structured well-owned sampled tested in prod. Observability for the Future™