SlideShare a Scribd company logo
Everything You Wanted to
Know About Distributed Tracing.
Strange Loop Conference
September 2019
Who Am I
● Hungai Amuhinda
● Twitter: @Hungai
● Nairobi, Kenya
● Work: Ajua, Infra Team
● Website: hungaikev.in
How to use this talk
● Please don’t just follow it blindly: There are often times when you will need
to do things differently.
● Software is all about trade-offs: Very few decisions are about right and
wrong.
● Try things for yourself: Why not make Friday afternoons a time to play and
experiment?
Software Evolution.
● Monolith
● On Prem
● Single Language
● Single Stack
● Virtual Machines
Software Evolution.
● Microservices
● Containers
● Multi Cloud/ Hybrid
● Polyglot
● Containers
● Serverless/ Cloud Functions
New Architectures/ New Challenges.
● Observability
● Deployment / Packaging
● Configuration Management
● Debugging
● Secrets Management
Meet: Distributed Tracing
“Distributed Tracing, also called distributed request tracing, is a method used to
profile and monitor applications, especially those built using a microservices
architecture. Distributed tracing helps pinpoint where failures occur and what
causes poor performance.”
Distributed Tracing - Terminology
Trace - a trace is a tree of spans that
follows the course of a request or
system from its source to its ultimate
destination.
Each trace is a narrative that tells the
requests story as it travels through the
system.
Distributed Tracing - Terminology
Span - are logical units of work in a
distributed system. They all have a
name, a start time, and a duration.
Each Span captures important data
points specific to the current process
handling the request.
Distributed Tracing - Terminology
Context Propagation:
Incoming
Request
Distributed Tracing - Terminology
Context Propagation:
Incoming
Request
trace-id = 123
parent-d = nil
span-id = 1
Distributed Tracing - Terminology
Context Propagation:
Incoming
Request
trace-id = 123
parent-d = nil
span-id = 1
Outbound
Request
trace-id = 123
parent-d = 1
span-id = 2
Distributed Tracing - Terminology
Tags & Logs: both annotate the span with some contextual information.
● Tags typically apply to the whole span, while logs represent some events that
happened during the span execution.
● A log always has a timestamp that falls within the span's start-end time interval.
● The tracing system does not explicitly track causality between logged events the
way it keeps track of causality relationships between spans, because it can be
inferred from the timestamps.
What questions can tracing help us answer?
Distributed Tracing:
● What services did a request pass through?
Distributed Tracing:
● What services did a request pass through?
● What occured in each service for a given request?
Distributed Tracing:
● What services did a request pass through?
● What occured in each service for a given request?
● Where did the error happen?
Distributed Tracing:
● What services did a request pass through?
● What occured in each service for a given request?
● Where did the error happen?
● Where are the bottlenecks?
Distributed Tracing:
● What services did a request pass through?
● What occured in each service for a given request?
● Where did the error happen?
● Where are the bottlenecks?
● What is the critical path for a request?
Distributed Tracing:
● What services did a request pass through?
● What occured in each service for a given request?
● Where did the error happen?
● Where are the bottlenecks?
● What is the critical path for a request?
● Who should I page?
If tracing is so good why isn’t everyone using it?
If tracing is so good why isn’t everyone using it?
● Not much education or not many publicized case studies on
the benefits.
If tracing is so good why isn’t everyone using it?
● Not much education or not many publicized case studies on
the benefits.
● Vendor Lock in is unacceptable: Instrumentation must be
decoupled from vendors
If tracing is so good why isn’t everyone using it?
● Not much education or not many publicized case studies on
the benefits.
● Vendor Lock in is unacceptable: Instrumentation must be
decoupled from vendors .
● Inconsistent APIs: Tracing semantics must not be language
dependent.
If tracing is so good why isn’t everyone using it?
● Not much education or not many publicized case studies on
the benefits.
● Vendor Lock in is unacceptable: Instrumentation must be
decoupled from vendors .
● Inconsistent APIs: Tracing semantics must not be language
dependent.
● Handoff woes: Tracing libs in Project X do not handoff to
tracing libs in Project Y.
Meet OpenTelemetry
OpenTelemetry
Open Telemetry is made up of an integrated set of APIs
and libraries as well as a collection mechanism via a agent
and collector. These components are used to generate,
collect, and describe telemetry about distributed
systems.
Problems OpenTelemetry solves:
● Vendor neutrality for tracing, monitoring and
logging
● Context Propagation.
OpenTelemetry (opentelemetry.io) Is:
● Single set of APIs for tracing and metrics collection.
● Standardized Context Propagation.
● Exporters for sending data to backend of choice.
● Collector for smart traces & metrics aggregation.
● Integrations with popular web, RPC and storage
frameworks.
OpenTelemetry (opentelemetry.io) Is:
Next major version of the OpenTracing and OpenCensus projects.
+ =
OpenTelemetry Roadmap:
+
Announcement: https://medium.com/opentracing/merging-opentracing-and-opencensus-f0fe9c7ca6f0
Roadmap: https://medium.com/opentracing/a-roadmap-to-convergence-b074e5815289
Tracing with OpenTelemetry - The Options
Agentless Using an Agent
OpenTelemetry: How to get Involved
Github: https://github.com/open-telemetry
Gitter: https://gitter.im/open-telemetry
Languages:
● .NET SDK
● GoLang SDK
● Java SDK
● JavaScript SDK
● Python SDK
● Ruby SIG
● Erlang/Elixir SDK
EXAMPLE
Create a Tracer
Create a Tracer
Tracers
Instrument
Instrument
Introducing Jaeger:
● Open source distributed tracing platform.
● Inspired by Google Dapper and OpenZipkin
● Created by Uber in 2015 and donated to CNCF in 2017.
● Compliant with both OpenTracing and OpenCensus.
● Supports multiple storage options (Cassandra, ElasticSearch, In-Memory)
● Compatible with Apache Kafka for backpressure management.
DEMO
obitech/micro-obs
Conclusion
● Tracing is crucial for understanding complex, microservices applications.
● Distributed tracing provides a base view of the system that can
drastically shorten feedback loops and the number of people involved
incidents.
● Tracing provides much more context, allowing an on call responder to
better understand the system and get further on their own before
involving more people.
Thank You
Twitter: @Hungai
Email: hungaikevin@gmail.com
Everything You wanted to Know About Distributed Tracing
Ad

More Related Content

What's hot (20)

Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
Distributed tracing 101
Distributed tracing 101Distributed tracing 101
Distributed tracing 101
Itiel Shwartz
 
OpenTelemetry For Developers
OpenTelemetry For DevelopersOpenTelemetry For Developers
OpenTelemetry For Developers
Kevin Brockhoff
 
Opentelemetry - From frontend to backend
Opentelemetry - From frontend to backendOpentelemetry - From frontend to backend
Opentelemetry - From frontend to backend
Sebastian Poxhofer
 
Observability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetryObservability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetry
DevOps.com
 
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfOSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
NETWAYS
 
Observability at Scale
Observability at Scale Observability at Scale
Observability at Scale
Knoldus Inc.
 
Observability vs APM vs Monitoring Comparison
Observability vs APM vs  Monitoring ComparisonObservability vs APM vs  Monitoring Comparison
Observability vs APM vs Monitoring Comparison
jeetendra mandal
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
Lucas Jellema
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
Grafana Labs
 
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.ioTHE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
DevOpsDays Tel Aviv
 
Meetup OpenTelemetry Intro
Meetup OpenTelemetry IntroMeetup OpenTelemetry Intro
Meetup OpenTelemetry Intro
DimitrisFinas1
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...
LibbySchulze
 
Prometheus and Grafana
Prometheus and GrafanaPrometheus and Grafana
Prometheus and Grafana
Lhouceine OUHAMZA
 
Observability & Datadog
Observability & DatadogObservability & Datadog
Observability & Datadog
JamesAnderson599331
 
Observability in the world of microservices
Observability in the world of microservicesObservability in the world of microservices
Observability in the world of microservices
Chandresh Pancholi
 
Grafana.pptx
Grafana.pptxGrafana.pptx
Grafana.pptx
Bhushan Rane
 
Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With Prometheus
Knoldus Inc.
 
Distributed tracing using open tracing & jaeger 2
Distributed tracing using open tracing & jaeger 2Distributed tracing using open tracing & jaeger 2
Distributed tracing using open tracing & jaeger 2
Chandresh Pancholi
 
How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...
How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...
How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...
Splunk
 
Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
Distributed tracing 101
Distributed tracing 101Distributed tracing 101
Distributed tracing 101
Itiel Shwartz
 
OpenTelemetry For Developers
OpenTelemetry For DevelopersOpenTelemetry For Developers
OpenTelemetry For Developers
Kevin Brockhoff
 
Opentelemetry - From frontend to backend
Opentelemetry - From frontend to backendOpentelemetry - From frontend to backend
Opentelemetry - From frontend to backend
Sebastian Poxhofer
 
Observability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetryObservability in Java: Getting Started with OpenTelemetry
Observability in Java: Getting Started with OpenTelemetry
DevOps.com
 
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdfOSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
OSMC 2022 | OpenTelemetry 101 by Dotan Horovit s.pdf
NETWAYS
 
Observability at Scale
Observability at Scale Observability at Scale
Observability at Scale
Knoldus Inc.
 
Observability vs APM vs Monitoring Comparison
Observability vs APM vs  Monitoring ComparisonObservability vs APM vs  Monitoring Comparison
Observability vs APM vs Monitoring Comparison
jeetendra mandal
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
Lucas Jellema
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
Grafana Labs
 
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.ioTHE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
DevOpsDays Tel Aviv
 
Meetup OpenTelemetry Intro
Meetup OpenTelemetry IntroMeetup OpenTelemetry Intro
Meetup OpenTelemetry Intro
DimitrisFinas1
 
Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...Intro to open source observability with grafana, prometheus, loki, and tempo(...
Intro to open source observability with grafana, prometheus, loki, and tempo(...
LibbySchulze
 
Observability in the world of microservices
Observability in the world of microservicesObservability in the world of microservices
Observability in the world of microservices
Chandresh Pancholi
 
Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With Prometheus
Knoldus Inc.
 
Distributed tracing using open tracing & jaeger 2
Distributed tracing using open tracing & jaeger 2Distributed tracing using open tracing & jaeger 2
Distributed tracing using open tracing & jaeger 2
Chandresh Pancholi
 
How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...
How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...
How to Move from Monitoring to Observability, On-Premises and in a Multi-Clou...
Splunk
 

Similar to Everything You wanted to Know About Distributed Tracing (20)

Introduction to the open rights group censorship monitoring project
Introduction to the open rights group censorship monitoring projectIntroduction to the open rights group censorship monitoring project
Introduction to the open rights group censorship monitoring project
Richard King
 
Blockade.io : One Click Browser Defense
Blockade.io : One Click Browser DefenseBlockade.io : One Click Browser Defense
Blockade.io : One Click Browser Defense
RiskIQ, Inc.
 
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
AgileNetwork
 
Go Observability (in practice)
Go Observability (in practice)Go Observability (in practice)
Go Observability (in practice)
Eran Levy
 
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
Paris Open Source Summit
 
Distributed tracing
Distributed tracingDistributed tracing
Distributed tracing
Puneeth Nanjundaswamy
 
The Final Frontier, Automating Dynamic Security Testing
The Final Frontier, Automating Dynamic Security TestingThe Final Frontier, Automating Dynamic Security Testing
The Final Frontier, Automating Dynamic Security Testing
Matt Tesauro
 
Tenants for Going at DevSecOps Speed - LASCON 2023
Tenants for Going at DevSecOps Speed - LASCON 2023Tenants for Going at DevSecOps Speed - LASCON 2023
Tenants for Going at DevSecOps Speed - LASCON 2023
Matt Tesauro
 
DevOps State of the Union 2015
DevOps State of the Union 2015DevOps State of the Union 2015
DevOps State of the Union 2015
Ernest Mueller
 
Observability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptxObservability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptx
OpsTree solutions
 
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
StormForge .io
 
The working architecture of NodeJS applications, Виктор Турский
The working architecture of NodeJS applications, Виктор ТурскийThe working architecture of NodeJS applications, Виктор Турский
The working architecture of NodeJS applications, Виктор Турский
Sigma Software
 
The working architecture of node js applications open tech week javascript ...
The working architecture of node js applications   open tech week javascript ...The working architecture of node js applications   open tech week javascript ...
The working architecture of node js applications open tech week javascript ...
Viktor Turskyi
 
BSIT3CD_Continuation of Cyber incident response (1).pdf
BSIT3CD_Continuation of Cyber incident response (1).pdfBSIT3CD_Continuation of Cyber incident response (1).pdf
BSIT3CD_Continuation of Cyber incident response (1).pdf
StevenJoeBiago
 
Orchestration, Automation and Virtualisation (OAV) in GÉANT
Orchestration, Automation and Virtualisation (OAV) in GÉANTOrchestration, Automation and Virtualisation (OAV) in GÉANT
Orchestration, Automation and Virtualisation (OAV) in GÉANT
CSUC - Consorci de Serveis Universitaris de Catalunya
 
Bringing it all together
Bringing it all togetherBringing it all together
Bringing it all together
MelissaMcKay15
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIE
DanBrown980551
 
Cynthia Wu: Satisfaction Not Guaranteed
Cynthia Wu: Satisfaction Not GuaranteedCynthia Wu: Satisfaction Not Guaranteed
Cynthia Wu: Satisfaction Not Guaranteed
Anna Royzman
 
Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...
Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...
Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...
Chris Hammerschmidt
 
Ahmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalerts
Ahmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalertsAhmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalerts
Ahmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalerts
Shekh Muenuddeen
 
Introduction to the open rights group censorship monitoring project
Introduction to the open rights group censorship monitoring projectIntroduction to the open rights group censorship monitoring project
Introduction to the open rights group censorship monitoring project
Richard King
 
Blockade.io : One Click Browser Defense
Blockade.io : One Click Browser DefenseBlockade.io : One Click Browser Defense
Blockade.io : One Click Browser Defense
RiskIQ, Inc.
 
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
Agile Gurugram 2023 | Observability for Modern Applications. How does it help...
AgileNetwork
 
Go Observability (in practice)
Go Observability (in practice)Go Observability (in practice)
Go Observability (in practice)
Eran Levy
 
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
Paris Open Source Summit
 
The Final Frontier, Automating Dynamic Security Testing
The Final Frontier, Automating Dynamic Security TestingThe Final Frontier, Automating Dynamic Security Testing
The Final Frontier, Automating Dynamic Security Testing
Matt Tesauro
 
Tenants for Going at DevSecOps Speed - LASCON 2023
Tenants for Going at DevSecOps Speed - LASCON 2023Tenants for Going at DevSecOps Speed - LASCON 2023
Tenants for Going at DevSecOps Speed - LASCON 2023
Matt Tesauro
 
DevOps State of the Union 2015
DevOps State of the Union 2015DevOps State of the Union 2015
DevOps State of the Union 2015
Ernest Mueller
 
Observability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptxObservability for Application Developers (1)-1.pptx
Observability for Application Developers (1)-1.pptx
OpsTree solutions
 
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
StormForge .io
 
The working architecture of NodeJS applications, Виктор Турский
The working architecture of NodeJS applications, Виктор ТурскийThe working architecture of NodeJS applications, Виктор Турский
The working architecture of NodeJS applications, Виктор Турский
Sigma Software
 
The working architecture of node js applications open tech week javascript ...
The working architecture of node js applications   open tech week javascript ...The working architecture of node js applications   open tech week javascript ...
The working architecture of node js applications open tech week javascript ...
Viktor Turskyi
 
BSIT3CD_Continuation of Cyber incident response (1).pdf
BSIT3CD_Continuation of Cyber incident response (1).pdfBSIT3CD_Continuation of Cyber incident response (1).pdf
BSIT3CD_Continuation of Cyber incident response (1).pdf
StevenJoeBiago
 
Bringing it all together
Bringing it all togetherBringing it all together
Bringing it all together
MelissaMcKay15
 
LF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIELF Energy Webinar: Introduction to TROLIE
LF Energy Webinar: Introduction to TROLIE
DanBrown980551
 
Cynthia Wu: Satisfaction Not Guaranteed
Cynthia Wu: Satisfaction Not GuaranteedCynthia Wu: Satisfaction Not Guaranteed
Cynthia Wu: Satisfaction Not Guaranteed
Anna Royzman
 
Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...
Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...
Machine Learning for (DF)IR with Velociraptor: From Setting Expectations to a...
Chris Hammerschmidt
 
Ahmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalerts
Ahmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalertsAhmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalerts
Ahmadabad mule soft_meetup_11_october_2020_errorhanlingandmonitoringalerts
Shekh Muenuddeen
 
Ad

Recently uploaded (20)

Societal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainabilitySocietal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainability
Jordi Cabot
 
How can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptxHow can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptx
laravinson24
 
FL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full VersionFL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full Version
tahirabibi60507
 
EASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License CodeEASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License Code
aneelaramzan63
 
Not So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java WebinarNot So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java Webinar
Tier1 app
 
Get & Download Wondershare Filmora Crack Latest [2025]
Get & Download Wondershare Filmora Crack Latest [2025]Get & Download Wondershare Filmora Crack Latest [2025]
Get & Download Wondershare Filmora Crack Latest [2025]
saniaaftab72555
 
Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025
Orangescrum
 
Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage Dashboards
Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage DashboardsAdobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage Dashboards
Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage Dashboards
BradBedford3
 
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
Andre Hora
 
PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025
mu394968
 
Download YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full ActivatedDownload YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full Activated
saniamalik72555
 
Exploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the FutureExploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the Future
ICS
 
Exploring Code Comprehension in Scientific Programming: Preliminary Insight...
Exploring Code Comprehension  in Scientific Programming:  Preliminary Insight...Exploring Code Comprehension  in Scientific Programming:  Preliminary Insight...
Exploring Code Comprehension in Scientific Programming: Preliminary Insight...
University of Hawai‘i at Mānoa
 
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdf
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdfMicrosoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdf
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdf
TechSoup
 
Adobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest VersionAdobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest Version
kashifyounis067
 
Top 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docxTop 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docx
Portli
 
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
Andre Hora
 
Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025
kashifyounis067
 
Maxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINKMaxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINK
younisnoman75
 
F-Secure Freedome VPN 2025 Crack Plus Activation New Version
F-Secure Freedome VPN 2025 Crack Plus Activation  New VersionF-Secure Freedome VPN 2025 Crack Plus Activation  New Version
F-Secure Freedome VPN 2025 Crack Plus Activation New Version
saimabibi60507
 
Societal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainabilitySocietal challenges of AI: biases, multilinguism and sustainability
Societal challenges of AI: biases, multilinguism and sustainability
Jordi Cabot
 
How can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptxHow can one start with crypto wallet development.pptx
How can one start with crypto wallet development.pptx
laravinson24
 
FL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full VersionFL Studio Producer Edition Crack 2025 Full Version
FL Studio Producer Edition Crack 2025 Full Version
tahirabibi60507
 
EASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License CodeEASEUS Partition Master Crack + License Code
EASEUS Partition Master Crack + License Code
aneelaramzan63
 
Not So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java WebinarNot So Common Memory Leaks in Java Webinar
Not So Common Memory Leaks in Java Webinar
Tier1 app
 
Get & Download Wondershare Filmora Crack Latest [2025]
Get & Download Wondershare Filmora Crack Latest [2025]Get & Download Wondershare Filmora Crack Latest [2025]
Get & Download Wondershare Filmora Crack Latest [2025]
saniaaftab72555
 
Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025Why Orangescrum Is a Game Changer for Construction Companies in 2025
Why Orangescrum Is a Game Changer for Construction Companies in 2025
Orangescrum
 
Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage Dashboards
Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage DashboardsAdobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage Dashboards
Adobe Marketo Engage Champion Deep Dive - SFDC CRM Synch V2 & Usage Dashboards
BradBedford3
 
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
What Do Contribution Guidelines Say About Software Testing? (MSR 2025)
Andre Hora
 
PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025PDF Reader Pro Crack Latest Version FREE Download 2025
PDF Reader Pro Crack Latest Version FREE Download 2025
mu394968
 
Download YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full ActivatedDownload YouTube By Click 2025 Free Full Activated
Download YouTube By Click 2025 Free Full Activated
saniamalik72555
 
Exploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the FutureExploring Wayland: A Modern Display Server for the Future
Exploring Wayland: A Modern Display Server for the Future
ICS
 
Exploring Code Comprehension in Scientific Programming: Preliminary Insight...
Exploring Code Comprehension  in Scientific Programming:  Preliminary Insight...Exploring Code Comprehension  in Scientific Programming:  Preliminary Insight...
Exploring Code Comprehension in Scientific Programming: Preliminary Insight...
University of Hawai‘i at Mānoa
 
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdf
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdfMicrosoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdf
Microsoft AI Nonprofit Use Cases and Live Demo_2025.04.30.pdf
TechSoup
 
Adobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest VersionAdobe Illustrator Crack FREE Download 2025 Latest Version
Adobe Illustrator Crack FREE Download 2025 Latest Version
kashifyounis067
 
Top 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docxTop 10 Client Portal Software Solutions for 2025.docx
Top 10 Client Portal Software Solutions for 2025.docx
Portli
 
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
TestMigrationsInPy: A Dataset of Test Migrations from Unittest to Pytest (MSR...
Andre Hora
 
Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025Adobe Master Collection CC Crack Advance Version 2025
Adobe Master Collection CC Crack Advance Version 2025
kashifyounis067
 
Maxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINKMaxon CINEMA 4D 2025 Crack FREE Download LINK
Maxon CINEMA 4D 2025 Crack FREE Download LINK
younisnoman75
 
F-Secure Freedome VPN 2025 Crack Plus Activation New Version
F-Secure Freedome VPN 2025 Crack Plus Activation  New VersionF-Secure Freedome VPN 2025 Crack Plus Activation  New Version
F-Secure Freedome VPN 2025 Crack Plus Activation New Version
saimabibi60507
 
Ad

Everything You wanted to Know About Distributed Tracing

  • 1. Everything You Wanted to Know About Distributed Tracing. Strange Loop Conference September 2019
  • 2. Who Am I ● Hungai Amuhinda ● Twitter: @Hungai ● Nairobi, Kenya ● Work: Ajua, Infra Team ● Website: hungaikev.in
  • 3. How to use this talk ● Please don’t just follow it blindly: There are often times when you will need to do things differently. ● Software is all about trade-offs: Very few decisions are about right and wrong. ● Try things for yourself: Why not make Friday afternoons a time to play and experiment?
  • 4. Software Evolution. ● Monolith ● On Prem ● Single Language ● Single Stack ● Virtual Machines
  • 5. Software Evolution. ● Microservices ● Containers ● Multi Cloud/ Hybrid ● Polyglot ● Containers ● Serverless/ Cloud Functions
  • 6. New Architectures/ New Challenges. ● Observability ● Deployment / Packaging ● Configuration Management ● Debugging ● Secrets Management
  • 7. Meet: Distributed Tracing “Distributed Tracing, also called distributed request tracing, is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.”
  • 8. Distributed Tracing - Terminology Trace - a trace is a tree of spans that follows the course of a request or system from its source to its ultimate destination. Each trace is a narrative that tells the requests story as it travels through the system.
  • 9. Distributed Tracing - Terminology Span - are logical units of work in a distributed system. They all have a name, a start time, and a duration. Each Span captures important data points specific to the current process handling the request.
  • 10. Distributed Tracing - Terminology Context Propagation: Incoming Request
  • 11. Distributed Tracing - Terminology Context Propagation: Incoming Request trace-id = 123 parent-d = nil span-id = 1
  • 12. Distributed Tracing - Terminology Context Propagation: Incoming Request trace-id = 123 parent-d = nil span-id = 1 Outbound Request trace-id = 123 parent-d = 1 span-id = 2
  • 13. Distributed Tracing - Terminology Tags & Logs: both annotate the span with some contextual information. ● Tags typically apply to the whole span, while logs represent some events that happened during the span execution. ● A log always has a timestamp that falls within the span's start-end time interval. ● The tracing system does not explicitly track causality between logged events the way it keeps track of causality relationships between spans, because it can be inferred from the timestamps.
  • 14. What questions can tracing help us answer?
  • 15. Distributed Tracing: ● What services did a request pass through?
  • 16. Distributed Tracing: ● What services did a request pass through? ● What occured in each service for a given request?
  • 17. Distributed Tracing: ● What services did a request pass through? ● What occured in each service for a given request? ● Where did the error happen?
  • 18. Distributed Tracing: ● What services did a request pass through? ● What occured in each service for a given request? ● Where did the error happen? ● Where are the bottlenecks?
  • 19. Distributed Tracing: ● What services did a request pass through? ● What occured in each service for a given request? ● Where did the error happen? ● Where are the bottlenecks? ● What is the critical path for a request?
  • 20. Distributed Tracing: ● What services did a request pass through? ● What occured in each service for a given request? ● Where did the error happen? ● Where are the bottlenecks? ● What is the critical path for a request? ● Who should I page?
  • 21. If tracing is so good why isn’t everyone using it?
  • 22. If tracing is so good why isn’t everyone using it? ● Not much education or not many publicized case studies on the benefits.
  • 23. If tracing is so good why isn’t everyone using it? ● Not much education or not many publicized case studies on the benefits. ● Vendor Lock in is unacceptable: Instrumentation must be decoupled from vendors
  • 24. If tracing is so good why isn’t everyone using it? ● Not much education or not many publicized case studies on the benefits. ● Vendor Lock in is unacceptable: Instrumentation must be decoupled from vendors . ● Inconsistent APIs: Tracing semantics must not be language dependent.
  • 25. If tracing is so good why isn’t everyone using it? ● Not much education or not many publicized case studies on the benefits. ● Vendor Lock in is unacceptable: Instrumentation must be decoupled from vendors . ● Inconsistent APIs: Tracing semantics must not be language dependent. ● Handoff woes: Tracing libs in Project X do not handoff to tracing libs in Project Y.
  • 27. OpenTelemetry Open Telemetry is made up of an integrated set of APIs and libraries as well as a collection mechanism via a agent and collector. These components are used to generate, collect, and describe telemetry about distributed systems. Problems OpenTelemetry solves: ● Vendor neutrality for tracing, monitoring and logging ● Context Propagation.
  • 28. OpenTelemetry (opentelemetry.io) Is: ● Single set of APIs for tracing and metrics collection. ● Standardized Context Propagation. ● Exporters for sending data to backend of choice. ● Collector for smart traces & metrics aggregation. ● Integrations with popular web, RPC and storage frameworks.
  • 29. OpenTelemetry (opentelemetry.io) Is: Next major version of the OpenTracing and OpenCensus projects. + =
  • 31. Tracing with OpenTelemetry - The Options Agentless Using an Agent
  • 32. OpenTelemetry: How to get Involved Github: https://github.com/open-telemetry Gitter: https://gitter.im/open-telemetry Languages: ● .NET SDK ● GoLang SDK ● Java SDK ● JavaScript SDK ● Python SDK ● Ruby SIG ● Erlang/Elixir SDK
  • 39. Introducing Jaeger: ● Open source distributed tracing platform. ● Inspired by Google Dapper and OpenZipkin ● Created by Uber in 2015 and donated to CNCF in 2017. ● Compliant with both OpenTracing and OpenCensus. ● Supports multiple storage options (Cassandra, ElasticSearch, In-Memory) ● Compatible with Apache Kafka for backpressure management.
  • 41. Conclusion ● Tracing is crucial for understanding complex, microservices applications. ● Distributed tracing provides a base view of the system that can drastically shorten feedback loops and the number of people involved incidents. ● Tracing provides much more context, allowing an on call responder to better understand the system and get further on their own before involving more people.