Tracing

for Java Developers

Philipp Krenn @xeraa

@xeraa
Developer
@xeraa
 Logs
Used by everyone?

@xeraa
 !
Logs Events
Structure & Context

@xeraa
{
"@timestamp": "2019-11-20T16:53:01.746Z",
"log.level": "INFO",
"message": "[philipp] failed to log in with password [***]",
"service.name": "gs-securing-web",
"process.thread.name": "http-nio-8080-exec-8",
"log.logger": "hello.AuthenticationEventListener",
"labels.event.category": "LOGIN_FAILURE",
"labels.user.name": "philipp",
"labels.source.ip": "0:0:0:0:0:0:0:1",
"labels.url.full": "/login"
}

@xeraa
Metrics
Used by most?

@xeraa
 Metrics
SLI, SLA, SLO

Four golden signals:

latency, traffic, errors, saturation

@xeraa
 Uptime
Used by many?

@xeraa
 Uptime
Synthetic / (pro)active monitoring

@xeraa
 Traces
Used by some?

@xeraa
 Traces
Application Performance Monitoring

Distributed Tracing

@xeraa
 ...the monitoring and
management of performance
and availability of software
applications.
— Wikipedia

@xeraa
 Simpler Times
https://www.kartar.net/2019/07/intro-to-distributed-tracing/

@xeraa
 Better Times
for Vendors
https://www.kartar.net/2019/07/intro-to-distributed-tracing/

@xeraa
 Goals
Transaction context

Reconstruct flow

Query & visualize transactions

@xeraa
Agents

@xeraa
 Agents
Language & framework specific

Detect start & end of request, capture errors

@xeraa
 Agents
Wrap operations in standard & known 3rd party
libraries

Extract additional information

@xeraa
 Agents
Little to no overhead

Trace & hook, not profile

@xeraa
 Agents
Live in / attach to app process
public static void main
premain
agentmain

https://www.javaadvent.com/2019/12/a-beginners-guide-to-java-agents.html

@xeraa
 premain

package sample;
public class SimpleAgent<?> {
public static void premain(String argument) {
System.out.println("Hello " + argument);
}
}

Premain-Class: sample.SimpleAgent

java -javaagent:/opt/agent.jar=World some.Program

 Instrumentation API

package sample;
public class ClassLoadingAgent {
public static void premain(String argument, Instrumentation instrumentation) {
instrumentation.addTransformer(new ClassFileTransformer() {
@Override
public byte[] transform(Module module, ClassLoader loader,
String name, Class<?> typeIfLoaded,
ProtectionDomain domain, byte[] buffer) {
System.out.println("Class was loaded: " + name);
return null;
}
});
}
}
 ByteBuddy

package sample;
public class ByteBuddySampleAgent {
public static void premain(String argument, Instrumentation instrumentation) {
new AgentBuilder.Default()
.type(ElementMatchers.any())
.transform((DynamicType.Builder<?> builder, TypeDescription type,
ClassLoader loader, JavaModule module) -> {
System.out.println("Class was loaded: " + name);
return builder;
}).installOn(instrumentation);
}
}
 Measuring Time

public class TimeMeasurementAdvice {

@Advice.OnMethodEnter
public static long enter() {
return System.currentTimeMillis();
}
@Advice.OnMethodExit(onThrowable = Throwable.class)
public static void exit(@Advice.Enter long start,
@Advice.Origin String origin) {
long executionTime = System.currentTimeMillis() - start;
System.out.println(origin + " took " + executionTime + " to execute");
}
}
 Measure Time through ByteBuddy

package sample;
public class ByteBuddyTimeMeasuringAgent {
public static void premain(String argument, Instrumentation instrumentation) {
Advice advice = Advice.to(TimeMeasurementAdvice.class);
new AgentBuilder.Default()
.type(ElementMatchers.isAnnotatedBy(MeasureTime.class))
.transform((DynamicType.Builder<?> builder, TypeDescription type,
ClassLoader loader, JavaModule module) -> {
return builder.visit(advice.on(ElementMatchers.isMethod());
}).installOn(instrumentation);
}
}
Automatic Instrumentation
Minimal app modification

@xeraa
— .NET:
app.UseElasticApm(Configuration);
— Node.js:
const apm = require('elastic-apm-node').start()
— Python:
apm = elasticapm.instrument()
— Ruby on Rails:
config.elastic_apm.service_name = 'MyApp'

@xeraa
 Automatic Instrumentation
Java

java -javaagent:/app/elastic-apm-agent.jar
-jar /app/app.jar

@xeraa
 Automatic Instrumentation
Configuration

environment:
- ELASTIC_APM_SERVICE_NAME=${ELASTIC_APM_SERVICE_NAME:-my-app}
- ELASTIC_APM_SERVER_URL=${ELASTIC_APM_SERVER_URL:-http://apm-server:8200}
- ELASTIC_APM_APPLICATION_PACKAGES=net.xeraa.my-app
- ELASTIC_APM_ENABLE_LOG_CORRELATION=true
- ELASTIC_APM_ENVIRONMENT=production

@xeraa
Tracing

@xeraa
 Terminology
Trace: record of a request, DAG

Span: named & timed single operation, optionally

nested

@xeraa
 Terminology
Span Context: IDing a request

Context Propagation: Attaching IDs

@xeraa
 Problem
uber-trace-id: 118c6c15301b9b3b3:56e66177e6e55a91: 18c6c15301b9b3b3:1

elastic-apm-traceparent: 00-f109f092a7d869fb4615784bacefcfd7-
5bf936f4fcde3af0-01

@xeraa
 OpenTracing
Vendor-neutral APIs for tracing

@xeraa
 Trace

https://opentracing.io/docs/overview/

@xeraa
 SpanContext

SpanContext:
- trace_id:"abc123"
- span_id:"xyz789"
- Baggage Items:
- special_id:"vsid1738"

@xeraa
API / Headers Only
No wire protocol
(in- or out-band)

@xeraa
 W3C Trace Context
Pass trace context information across systems

@xeraa
 Trace Context Propagation

traceparent:
00- // Version
0af7651916cd43dd8448eb211c80319c- // Trace ID
b7ad6b7169203331- // Parent ID
01 // Sampling flags

CorrelationContext / baggage supported

@xeraa
 OpenTelemetry
OSS, beta, CNCF sandbox

Traces, metrics, logs

Supersedes OpenTracing + OpenCensus

@xeraa
 OpenTelemetry
Unification of instrumentation and pillars

Vendor neutral but wide support

Google, Splunk, Lightstep, Honeycomb, Elastic,...

@xeraa
 OpenTracing marketed itself as a
standard from the onset, never even
released a 1.0 version of its java
impl before canceling the project.
OpenCensus also never made 1.0 in its
years before canceling itself. [...]
— https://twitter.com/adrianfcole/status/1223778238469566464

@xeraa
3 Ways to OpenTelemetry
Integration
Vendors can implement one or more

@xeraa
OpenTelemetry Protocol (OTLP)
Protocol for OpenTelemetry data exchange
Supports gRPC and HTTP over Protobuf
Experimental HTTP over JSON

@xeraa
 OpenTelemetry for Java
github.com/open-telemetry/opentelemetry-java:
standalone library providing OTel support

github.com/open-telemetry/opentelemetry-java-
instrumentation: auto-instrumentation agent,
relying on opentelemetry-java

@xeraa
 opentelemetry-java High-Level Components

— API: opentelemetry-java/api
— SDK (API implementation): opentelemetry-java/sdk
— Extensions for the API: opentelemetry-java/extensions
— Extensions for the SDK: opentelemetry-java/sdk-
extensions
— Compatibility layers (shim) opencensus-shim &
opentracing-shim

@xeraa
 Status
opentelemetry-java 1.5.0 (2021/08), GA since
2021/02

opentelemetry-java-instrumentation 1.4.1 (2021/07),

GA since 2021/03

@xeraa
Demo

@xeraa
Elastic Agent & Jaeger

@xeraa
Elastic Agent & Jaeger

@xeraa
OpenTelemetry

@xeraa
 Auto Instrumentation

java -javaagent:/opt/opentelemetry-javaagent-all.jar \
-jar myapp.jar

@xeraa
 Manual Instrumentation

Tracer tracer =
OpenTelemetry.getGlobalTracer("instrumentation-library-name", "1.0.0");

Span span = tracer.spanBuilder("my span").startSpan();

try (Scope scope = span.makeCurrent()) {
// your use case
...
} catch (Throwable t) {
span.setStatus(StatusCode.ERROR, "Change it to your error message");
} finally {
span.end(); // closing the scope does not end the span; has to be done manually
}

@xeraa
 Data in a Trace
Just another index

Elastic Common Schema (ECS)

@xeraa
 Scale & Cleanup: ILM (modified)

{
"policy" : {
"phases" : {
"hot" : {
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "30d"
} } },
"delete" : {
"min_age" : "30d",
"actions" : {
"delete" : { }
Conclusion

@xeraa
Tools
Logs

Metrics

Uptime

Traces

@xeraa
Distributed Tracing
Why & How

@xeraa
OpenTelemetry
Why & How

@xeraa
Battle on the Backend
Story, search, visualize, manage

@xeraa
Tracing
for Java Developers

Philipp Krenn @xeraa

@xeraa