The Last Pickle: Distributed Tracing from Application to Database
4 likes1,105 views
This document discusses the integration of Zipkin tracing with Apache Cassandra to enhance the tracing and profiling of microservices. It outlines the challenges of scaling and reproducing problems in distributed systems and provides a solution for implementing Zipkin tracing in Cassandra environments. It includes references to relevant resources, patches, and classes for effective implementation.
The Last Pickle: Distributed Tracing from Application to Database
- 3. ABOUT THE LAST PICKLE WORK WITH CLIENTS TO DELIVER AND IMPROVE APACHE CASSANDRA BASED SOLUTIONS BASED IN USA, NEW ZEALAND, AUSTRALIA
- 4. microservices - devOps - distributed tracing zipkin zipkin & cassandra
- 5. Apache Cassandra the data platform de jure for the next evolution of software services an enterprise moving ever towards microservices and BASE architectures the missing piece for many is tracing and profiling difficult to reproduce problems
- 6. SCALING DATA Apache Cassandra data platform de jure next evolution of software services
- 7. SCALING PEOPLE an enterprise moving ever towards microservices and BASE architectures
- 10. ARCHITECTURAL SAFETY zero exceptions spot problems first stable master no user left out plug-n-play services
- 11. tracing and profiling difficult to reproduce problems
- 12. ZIPKIN an implementation of Google's Dapper paper
- 13. SEARCH TRACES
- 17. CLIENT | SERVER CS --> --> SR <-- SS CR <--
- 19. SIMPLE HTTP CALL
- 20. SIMPLE HTTP CALL
- 21. SIMPLE C* CALL
- 22. SIMPLE C* CALL
- 23. HTTP CALL PASSING THROUGH HEADERS
- 24. TRACING IN C*
- 27. CO-ORDINATOR NODE REPLICA NODE --> beginSession(..) trace(..) trace(..) --> initialiseMessage(..) trace(..) trace(..) <-- trace(..) endSession(..) <--
- 28. ZIPKIN IN C* visualisation detailed timings hierarchy and asynchronisity zero tracing overhead
- 30. Two classes to override then run enabling new tracing public class ZipkinTracing extends Tracing {..} public class ZipkinTraceState extends TraceState {..} bin/cassandra -Dcassandra.custom_tracing_class=..ZipkinTracing
- 31. ZIPKIN ACROSS C*
- 32. ZIPKIN ACROSS C* CO-ORDINATOR NODE REPLICA NODE --> beginSession(..) trace(..) trace(..) (zipkin headers) --> initialiseMessage(..) trace(..) trace(..) <-- trace(..) endSession(..) <--
- 33. ZIPKIN ACROSS C*
- 35. ZIPKIN INTO C*
- 36. ZIPKIN INTO C*
- 38. TO THE RESCUE
- 40. HTTP CALL PASSING THROUGH HEADERS
- 41. C* CALL USING CUSTOM PAYLOAD
- 42. enable zipkin tracing the custom payload handlerand bin/cassandra -Dcassandra.custom_tracing_class=..ZipkinTracing -Dcassandra.custom_query_handler_class=..CustomPayloadMirroringQueryHandler
- 43. the patch? src/java/org/apache/cassandra/net/MessageOut.java | 7 +------ src/java/org/apache/cassandra/net/OutboundTcpConnection.java | 4 +++- src/java/org/apache/cassandra/service/QueryState.java | 12 ++++++++++-- src/java/org/apache/cassandra/transport/messages/ExecuteMessage.java| 2 +- 4 file changed, 15 insertion(+), 10 deletion(-)
- 45. THANKS Zipkin – https://github.com/openzipkin/zipkin Brave (zipkin java instrumentation) – https://github.com/openzipkin/brave C* patch for pluggable tracing – https://issues.apache.org/jira/browse/CASSANDRA-10392 Zipkin Cassandra implementation – https://github.com/thelastpickle/cassandra-zipkin-tracing Google's Dapper paper – http://research.google.com/pubs/pub36356.html C* custom payloads – https://issues.apache.org/jira/browse/CASSANDRA-8553 – https://datastax.github.io/java-driver/2.2.0-rc2/features/custom_payloads/