Combining Logs, Metrics, and Traces for Unified Observability
0 likes•1,341 views
The document discusses the complexities of unified observability in cloud-native environments, emphasizing the need for centralized logging, metrics, and application performance monitoring (APM). It highlights the evolution of monitoring architectures and the transition from traditional licensing models to more scalable options that integrate various data sources. Additionally, it underlines the importance of a unified data layer and machine learning for actionable insights in managing application performance and operational efficiency.
Combining Logs, Metrics, and Traces for Unified Observability
- 1. Tanya Bragin Senior Director, Product Management April 2020 Logs, Metrics, and APM for Unified Observability
- 3. Higher resource utilization increases monitoring complexity • Orchestration/Hypervisor • Dynamic/ephemeral jobs • You can no longer "point" to where that job lives Shift to cloud-native yields maintainable code, with costs • Traditional licensing models don't scale as well as your applications • Hurdles with autoscaling Monitoring Complexity Hardware & software trends are evolving in tandem Evolving Architectures ~↑ Monitoring Complexity
- 4. Applications VMs/Containers Other DBs, Services & Middleware Orchestration InfrastructureUptime Metrics Logs Network Network APM Metrics APM Logs APM APM Metrics Logs Network Metrics Logs Network APM Network APM Uptime
- 5. Development & DevOps Teams Log Monitoring Team Interface status Flows (Netflow, sFlow, IPFIX) Real traffic (packet analysis) Network Tool Infra Monitoring Team Web Logs App Logs Database Logs Container Logs Middleware Logs Log Tool Network Monitoring Team Real User Monitoring Txn Perf Monitoring Distributed Tracing Uptime Response Time APM & Uptime Tools Container Metrics Host Metrics Database Metics Network Metrics Storage Metrics Metrics Tool Status Quo: Siloed Collection of Tools
- 6. How many tools does your org currently use for monitoring your systems?
- 7. APM & Uptime NetworkMetricsLogs Elastic Approach to Observability Interface status Flows (Netflow, sFlow, IPFIX) Real traffic (packet analysis) Web Logs App Logs Database Logs Container Logs Middleware Logs Real User Monitoring Txn Perf Monitoring Distributed Tracing Uptime Response time Container Metrics Host Metrics Database Metics Network Metrics Storage Metrics Elastic Common Schema
- 8. Unified User Interface Same UI for KPI summaries and root-cause analysis
- 9. Unified Data Layer with Common Schema Open data keeps your data out of silos and delivers maximum business value • Ship from anywhere — and correlate across your data these sources • The data is yours — no API rate limiting, no data black boxes • Cloud native scale — no constraints on dimensions and cardinality
- 10. Correlate all data sources with unified machine learning and anomaly detection Unified Machine Learning and Alerting
- 11. APM & Uptime NetworkMetricsLogs Elastic Approach to Observability Interface status Flows (Netflow, sFlow, IPFIX) Real traffic (packet analysis) Web Logs App Logs Database Logs Container Logs Middleware Logs Real User Monitoring Txn Perf Monitoring Distributed Tracing Uptime Response time Container Metrics Host Metrics Database Metics Network Metrics Storage Metrics Elastic Common Schema
- 12. Elastic Stack for logs
- 13. Adopt an open approach to centralized logging Turnkey data ingestion, intuitive search interface
- 14. Make logs actionable with machine learning Improve analyst efficiency: 10,000 foot view to a single log line
- 15. Turn log events into intelligence Real-time dashboards based on log data, at scale
- 16. Meet audit requirements with log lifecycle management Index lifecycle management Policy based data management that optimizes your cluster behind the scenes Hot. Warm. Cold. Frozen.Log archival and re-hydration Robust snapshot management via API or Snapshot Management UI Cold storage with online search Specialized indices for efficient long-term retention of logs You’re in control of how your data is tiered
- 17. Elastic Stack for metrics
- 18. Elastic Stack as a Metrics Store BKD trees Data structures optimized for numerical time series analysis. Columnar storage Structured data storage, resulting in compact storage and faster analytics Rollups and Index Lifecycle Management Aggregate older data into bigger time buckets Aggregations framework Analytics features to slice and dice data along various dimensions 2012 2016 2014 2018 Prometheus support Support for ingesting data from Prometheus exporters and servers 2019 Improved support for histograms Dedicated histogram data type in Elasticsearch 2020
- 19. Turnkey data on-boarding 100s of data sources at your fingertips
- 20. Turn metrics into intelligence Flexible time-series analytics and data visualization
- 21. Make logs more valuable with metrics From KPIs to logs
- 22. Combine SLA monitoring with logs Easy-to-consume interface, unified with the rest of observability data
- 23. Make your data actionable with alerting In-context alert creation
- 24. Make your data actionable with alerting In-context alert creation
- 25. Make your data actionable with alerting In-context alert creation
- 26. Elastic Stack for network and APM
- 27. 27 Elastic for Network Packetbeat joins Elastic Added real-time packet analytics to the stack 2016 2017 Elastic adds support for SNMP Network device and interface health information 2019 2018 Elastic adds support for Netflow Out of the box support for flow-level visibility From interfaces to flows to packets Add Network view in the SIEM app Network activity view relevant for security and observability alike
- 28. 28 Elastic APM Elastic joins forces with Opbeat A next-generation APM solution designed for developers 2017 2018 Distributed tracing Auto-instrumentation and support for OpenTracing, W3C Trace Context header 2020 2019 Elastic APM GA & more agents Agents for Python, Node.js, Ruby, Javascript; Real User Monitoring, Java, … Enterprise-ready free and open APM ● Java ● .NET ● Node.js ● Javascript Language Support ● Python ● Ruby ● Go ● PHP (in dev) • Turnkey agents • Auto-instrumentation for common frameworks • Designed to be lightweight
- 29. 29 Elastic APM Elastic joins forces with Opbeat A next-generation APM solution designed for developers 2017 2018 Distributed tracing Auto-instrumentation and support for OpenTracing, W3C Trace Context header 2020 2019 Elastic APM GA & more agents Agents for Python, Node.js, Ruby, Javascript; Real User Monitoring, Java, … Service Map, annotations Fully features user interface for navigating APM data Enterprise-ready free and open APM
- 30. Avoid lock-in with open source APM agents Support for open standards - Jaeger, OpenTracing, OpenMetrics, W3C Trace context
- 31. Track transactions from browser to backend End-to-end distributed tracing
- 32. Reduce MTTR by streamlining analyst workflow Navigate traces, metrics, and logs in one UI for faster issue resolution
- 33. Understand your dependencies in real time Dependency mapping
- 34. Get more value from your trace data Flexible data retention for detailed traces per application class • Stop throwing away valuable traces before they were analyzed • Apply machine learning to detailed trace data to gain insights • Set up data retention policies per application class to contain costs
- 35. 35 Demo
- 36. What now? Try it yourself!
- 37. Next up: Workplace Search