The RED Method: How to monitoring your microservices.
1 like1,536 views
The document discusses the RED method for monitoring services, focusing on key metrics: Requests (rate, errors, duration), and the USE method for resource monitoring (utilization, saturation, errors). It provides Prometheus implementations and examples for tracking these metrics across CPU, memory, disk, and network resources. Additionally, it mentions the four golden signals of monitoring: latency, traffic, errors, and saturation, alongside references for further reading.
![CPU USE in Prometheus
CPU Utilisation:
1 - avg(rate(node_cpu{job="default/node-exporter",mode="idle"}[1m]))
CPU Saturation:
sum(node_load1{job="default/node-exporter"})
/
sum(node:node_num_cpu:sum)](https://image.slidesharecdn.com/theredmethod-181022165523/85/The-RED-Method-How-to-monitoring-your-microservices-8-320.jpg)
![Memory USE in Prometheus
Memory Utilisation:
1 - sum(
node_memory_MemFree{job=“…”} +
node_memory_Cached{job=“…”} +
node_memory_Buffers{job=“…”}
)
/ sum(node_memory_MemTotal{job=“…”})
Memory Saturation:
1e3 * sum(
rate(node_vmstat_pgpgin{job=“…”}[1m]) +
rate(node_vmstat_pgpgout{job=“…”}[1m]))
)](https://image.slidesharecdn.com/theredmethod-181022165523/85/The-RED-Method-How-to-monitoring-your-microservices-9-320.jpg)
![Prometheus Implementation
import (
“github.com/prometheus/client_golang/prometheus"
)
var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "request_duration_seconds",
Help: "Time (in seconds) spent serving HTTP requests.",
Buckets: prometheus.DefBuckets,
}, []string{"method", "route", "status_code"})
func init() {
prometheus.MustRegister(requestDuration)
}](https://image.slidesharecdn.com/theredmethod-181022165523/85/The-RED-Method-How-to-monitoring-your-microservices-16-320.jpg)
![Easy to query
Rate:
sum(rate(request_duration_seconds_count{job=“…”}[1m]))
Errors:
sum(rate(request_duration_seconds_count{job=“…”,
status_code!~”2..”}[1m]))
Duration:
histogram_quantile(0.99,
sum(rate(request_duration_seconds_bucket{job=“…}[1m])) by (le))](https://image.slidesharecdn.com/theredmethod-181022165523/85/The-RED-Method-How-to-monitoring-your-microservices-18-320.jpg)
The RED Method: How to monitoring your microservices.
- 1. The RED Method Patterns for instrumentation & monitoring. Tom Wilkie [email protected] @tom_wilkie github.com/tomwilkie
- 3. Introduction Why does this matter? The USE Method Utilisation, Saturation, Errors The RED Method Requests Rate, Errors, Duration.. The Four Golden Signals RED + Saturation
- 4. Introduction
- 6. For every resource, monitor: • Utilisation: % time that the resource was busy • Saturation: amount of work resource has to do, often queue length • Errors: the count of error events http://www.brendangregg.com/usemethod.html
- 7. Utilisation Saturation Errors CPU ✔ ✔ ✔ Memory ✔ ✔ ✔ Disk ✔ ✔ ✔ Network ✔ ✔ ✖
- 8. CPU USE in Prometheus CPU Utilisation: 1 - avg(rate(node_cpu{job="default/node-exporter",mode="idle"}[1m])) CPU Saturation: sum(node_load1{job="default/node-exporter"}) / sum(node:node_num_cpu:sum)
- 9. Memory USE in Prometheus Memory Utilisation: 1 - sum( node_memory_MemFree{job=“…”} + node_memory_Cached{job=“…”} + node_memory_Buffers{job=“…”} ) / sum(node_memory_MemTotal{job=“…”}) Memory Saturation: 1e3 * sum( rate(node_vmstat_pgpgin{job=“…”}[1m]) + rate(node_vmstat_pgpgout{job=“…”}[1m])) )
- 10. Interesting / Hard Cases • CPU Errors, Memory Errors • Hard Disk Errors! • Disk Capacity vs Disk IO • Network Utilisation • Interconnects
- 11. Demo Time
- 12. More Details • “The USE Method” - Brendan Gregg • Kubernetes Mixin - https://github.com/grafana/jsonnet-libs
- 13. The RED Method
- 14. For every service, monitor request: • Rate - number of requests per second • Errors - the number of those requests that are failing • Duration - the amount of time those requests take The RED Method
- 16. Prometheus Implementation import ( “github.com/prometheus/client_golang/prometheus" ) var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{ Name: "request_duration_seconds", Help: "Time (in seconds) spent serving HTTP requests.", Buckets: prometheus.DefBuckets, }, []string{"method", "route", "status_code"}) func init() { prometheus.MustRegister(requestDuration) }
- 17. func wrap(h http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { m := httpsnoop.CaptureMetrics(h, w, r) requestDuration.WithLabelValues(r.Method, r.URL.Path, strconv.Itoa(m.Code)).Observe(m.Duration.Seconds()) }) } func server(addr string) { http.Handle("/metrics", prometheus.Handler()) http.Handle("/greeter", wrap(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ... })) } Prometheus Implementation
- 19. Demo Time
- 20. DAG of Services
- 23. More Details • “Monitoring Microservices” - Weaveworks (slides) • "The RED Method: key metrics for microservices architecture” - Weaveworks • “Monitoring and Observability with USE and RED” - VividCortex • “RED Method for Prometheus – 3 Key Metrics for Monitoring” - Rancher Labs • “Logs and Metrics” - Cindy Sridharan • "Logging v. instrumentation”, “Go best practices, six years in” - Peter Bourgon
- 26. The Four Golden Signals For each service, monitor: • Latency - time taken to serve a request • Traffic - how much demand is places on your system • Errors - rate or requests that are failing • Saturation - how “full” your services is
- 28. Demo Time
- 29. More Details • “The Four Golden Signals” - The Google SRE Book • “How to Monitor the SRE Golden Signals” - Steve Mushero
- 30. Summary
- 32. Tom Wilkie VP Product, Grafana Labs Previously: Kausal, Weaveworks, Google, Acunu, Xensource Twitter: @tom_wilkie Email: [email protected]
- 33. + Grafana Cloud is a hosted and fully managed SaaS metrics platform that helps Ops and Dev teams using Grafana to understand the behavior of their applications and infrastructure Grafana Cloud allows users to provision and manage the best open source observability tools - Grafana and Prometheus - all through a simple UI and single API. What is Grafana Cloud? Store, visualize and alert without the headache of scaling or managing your own monitoring stack. Your complete, fully managed, hosted metrics platform. Grafana Cloud: