Monitoring and Instrumentation Strategies: Tips and Best Practices - AppSphere16
3 likes3,592 views
This document discusses different ways to instrument software applications for monitoring purposes. It begins with definitions of instrumentation and telemetry. It then discusses the different types of data that can be collected through instrumentation like metrics and events. It describes how to instrument various parts of applications including the frontend like browsers and mobile apps as well as the backend for languages like Java, .NET, Node.js and PHP. It provides best practices for instrumentation and logging. Finally, it discusses challenges with correlating data across distributed systems and some open source options and solutions for transaction correlation.
![Monkey patching
Wikipedia relevant definition:
In Ruby,[3] Python,[4] and many other dynamic
programming languages... dynamic modifications of a
class or module at runtime, motivated by the intent to
patch existing third-party code as a workaround to a
bug or feature which does not act as desired
Disclaimer : Can be very dangerous, hard to maintain
• Replace methods / attributes / functions at runtime
• Apply a patch at runtime to the objects in memory,
instead of the source code on disk;](https://image.slidesharecdn.com/t8455-kowall-161123000824/85/Monitoring-and-Instrumentation-Strategies-Tips-and-Best-Practices-AppSphere16-23-320.jpg)
![Correlation in logs
• Log every transaction segment
• Persist a GUID or transaction ID
• This is very difficult in large teams
• Inefficient to analyze and pull metrics from logs
• Doesn’t work unless you own the code
[code]
PERF,2013-04-03 11:29:52.640,external,0x123456,NA,service1,MyAPP,jimmy,NA,336,NA,NA
INFO,2013-04-03 11:29:53.189,internal,789012,0x123456,service2,TheirApp,jimmy,NA,174,NA,NA
INFO,2013-04-03 11:29:52.892,internal,345678,789012,service3,TheirApp,jimmy,NA,163,NA,NA
[/code]](https://image.slidesharecdn.com/t8455-kowall-161123000824/85/Monitoring-and-Instrumentation-Strategies-Tips-and-Best-Practices-AppSphere16-35-320.jpg)
![Transaction correlation and logs!
• Many integrations across APM
and Log vendors
• Can add correlation in code
and use any log tool
– ex: [%X{AD.requestGUID}]
• We auto inject and correlate
(one platform)](https://image.slidesharecdn.com/t8455-kowall-161123000824/85/Monitoring-and-Instrumentation-Strategies-Tips-and-Best-Practices-AppSphere16-36-320.jpg)
![[NDC17] Kubernetes로 개발서버 간단히 찍어내기](https://cdn.slidesharecdn.com/ss_thumbnails/ndc-170529041601-thumbnail.jpg?width=560&fit=bounds)
![[오픈소스컨설팅] 프로메테우스 모니터링 살펴보고 구성하기](https://cdn.slidesharecdn.com/ss_thumbnails/oscprometheus-190422050231-thumbnail.jpg?width=560&fit=bounds)
![[Spring Camp 2018] 11번가 Spring Cloud 기반 MSA로의 전환 : 지난 1년간의 이야기](https://cdn.slidesharecdn.com/ss_thumbnails/201804springcamp11stmsafinalpubslideshare-180527051608-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Monitoring and Instrumentation Strategies: Tips and Best Practices - AppSphere16 (20)
Ad
More from AppDynamics (20)
Recently uploaded (20)
Monitoring and Instrumentation Strategies: Tips and Best Practices - AppSphere16
- 1. The differing ways to monitor and instrument Jonah Kowall VP Market Development and Insights Twitter : @jkowall
- 2. APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 2 Notice The information and materials included in this presentation (collectively, the “Materials”) are the proprietary information of AppDynamics, Inc. (“AppDynamics” or the “Company”). No part of the Materials may be reproduced, distributed, communicated or displayed in any form or by any means, or used to make any derivative work, without prior written permission from AppDynamics. The Materials may contain product roadmap information of AppDynamics. AppDynamics reserves the right to change any product roadmap information at any time, for any reason and without notice. This information is intended to outline AppDynamics' general product direction, it is not a guarantee of future product features, and it should not be relied on in making a purchasing decision. The development, release, and timing of any features or functionality described for AppDynamics' products remains at AppDynamics' sole discretion. AppDynamics reserves the right to change any planned features at any time before making them generally available as well as never making them generally available. All third-party trademarks, including names, logos and brands, referenced by AppDynamics in this presentation are property of their respective owners. All references to third-party trademarks are for identification purposes only and shall be considered nominative fair use under trademark law. © 2016 AppDynamics, Inc. All rights reserved.
- 3. Jonah Kowall’s background • 23 years in IT • Infrastructure and Operations enterprises and startups (17 yrs) – Security - CISSP, CISA, PCI – Started one of the first content filtering companies • Head of global monitoring at Thomson Reuters • Head of IT Operations at MFG.com – Bezos Expeditions • Gartner Research VP 3.5 years • Strategy AppDynamics 1.5 years
- 4. Agenda • Introduction to instrumentation • Instrumentation defined • Priorities in instrumentation • Technical instrumentation of frontends (Browser and Mobile) • Technical instrumentation of backend (Java, .NET, Python, Node.js, PHP) • Logging best practices • How transaction correlation options (AppDynamics and Open Source)
- 5. Definitions Instrumentation “The design, construction, and provision of instruments for measurement, control, etc; the state of being equipped with or controlled by such instruments collectively.” https://en.wikipedia.org/wiki/Telemetry Telemetry “Automated communications process by which measurements are made and other data collected at remote or inaccessible points and transmitted to receiving equipment for monitoring.” https://en.wikipedia.org/wiki/Instrumentation
- 6. Software instrumentation data types • Metrics – Key value pairs – Numeric values – Time series • Events – Informational – Errors – Critical events
- 7. Use cases for data types • Metrics – Average, peak – Percentage – Correlation to metrics and events • Events – Search – Parse – Correlate to metrics
- 8. Software instrumentation • Logging – Supplied by vendor – Created by developers – Not easily controlled • Push Collection – Attach and extract – Software agent or network tap • Pull Collection – Polling APIs - HTTP, SNMP, WMI
- 10. Priorities for instrumentation • Business • Application • Services • Infrastructure Events and metrics
- 11. Uplevel the conversation • Understand the customer – Internally and externally • Requirements should be gathered across business and IT teams • Responsibility for definition of monitoring should be shared
- 12. Business metrics and KPIs • Customer metrics – Conversion between products – Loyalty and retention (churn) – Usage metrics (feature and product) • Sales / marketing metrics – Revenue – Cost of customer acquisition – User flows through applications
- 13. Technical metrics and KPIs • End to end performance – User through transaction hops – Error isolation • End user experience – Client side errors – Latency per element (page or app) + 3rd party – Client side DNS • Application component performance – Metrics from app server – Metrics from code – Queries – Errors • Intra application component performance
- 14. Use cases for business and technical data • Usage • Problem identification - MTTI • Problem resolution - MTTR • User satisfaction • Usability • Performance • Change analysis – A/B testing – data center moves – technology changes
- 18. Simulating users • Synthetic transactions are good for • SLAs • Availability • Baseline performance • DNS • SSL • But if you try to use it as a barometer you will fail
- 19. Backend instrumentation of Java and .NET
- 20. Java instrumentation JSR-163 (JavaTM Platform Profiling Architecture) added in Java 1.5 Overloads the default behavior of Java to allow hooks into code for many use cases Since JDK 1.6, for the Oracle HotSpot JVM, a javaagent may be dynamically attached to a running JVM by specifying the process-id (pid).
- 21. .NET Instrumentation ● Profiling API loaded into the same process as the application process that is being profiled. ● Callback interface (ICorProfilerCallback in the .NET Framework version 1.0 and 1.1, ICorProfilerCallback2 in version 2.0 and later) ● CLR calls the methods in that interface to notify the .NET agent of events in the profiled process ● Profiler can also call into the CLR by using the methods in the ICorProfilerInfo and ICorProfilerInfo2 interfaces to obtain information about the state of the profiled application ● Callbacks are used to inject MSIL (Microsoft Intermediate Language) bytecode into existing application code for instrumentation.
- 22. Backend instrumentation of interpreted languages
- 23. Monkey patching Wikipedia relevant definition: In Ruby,[3] Python,[4] and many other dynamic programming languages... dynamic modifications of a class or module at runtime, motivated by the intent to patch existing third-party code as a workaround to a bug or feature which does not act as desired Disclaimer : Can be very dangerous, hard to maintain • Replace methods / attributes / functions at runtime • Apply a patch at runtime to the objects in memory, instead of the source code on disk;
- 24. PHP instrumentation Zend callback methods zend_execute(…), zend_execute_internal(…) and zend_compile_file(…) so that it can wrap the original implementations with instrumentation code. Handles state changes and new web server initialization (which are PHP instances)
- 25. Node.js instrumentation ● Wrap methods using before, after and around aspect interceptors. ● Callback along with after, before and around aspect interceptor. ● Notifications when asynchronous calls are complete.
- 28. Correlation in end to end APM
- 29. Correlation in asynchronous calls (headache)
- 30. Correlation in Open Source • OpenZipkin is the most advanced (backed by Pivotal) – New instrumentations (Java, Scala, JavaScript, Python, Ruby, C#, Go) – Requires code changes – No async support, no overhead controls (dependant on how implemented in code)
- 31. Future of correlation in Open Source • OpenTracing an open standard API for instrumentation – Vendors must implement specific code per instrumentation library which doesn’t scale – Must add code manually – No overhead controls • PivotTracing runs distributed traces on demand backed by Brown University – Currently has been written into forks of • Apache Hadoop (YARN, MapReduce, HDFS) • HBase • ZooKeeper
- 32. Open Source is starting to merge (Zipkin + Spigo) Spigo created by Adrian Cockroft for Simulation and Visualization of a complex App Topology Can now feed Zipkin traces into Spigo and visualize response times for a trace in a topology (still manual using 3erd party tool, and it’s missing any kind of real UI)
- 34. Logging best practices • Logs are not transaction records • Log errors and exceptions • Easily parsed (JSON) • Time (long), Source • Write your own identifiers for each statement logged (or instrument and inject) • Think about security implications (plain text, on disk, syslog are all insecure) • Keep log statements small (thanks Java, .NET…) • Every log statement introduces overhead, so don’t overdo it • Exception logs create even more overhead, so fix them • Do not try to use log tools as metric stores
- 35. Correlation in logs • Log every transaction segment • Persist a GUID or transaction ID • This is very difficult in large teams • Inefficient to analyze and pull metrics from logs • Doesn’t work unless you own the code [code] PERF,2013-04-03 11:29:52.640,external,0x123456,NA,service1,MyAPP,jimmy,NA,336,NA,NA INFO,2013-04-03 11:29:53.189,internal,789012,0x123456,service2,TheirApp,jimmy,NA,174,NA,NA INFO,2013-04-03 11:29:52.892,internal,345678,789012,service3,TheirApp,jimmy,NA,163,NA,NA [/code]
- 36. Transaction correlation and logs! • Many integrations across APM and Log vendors • Can add correlation in code and use any log tool – ex: [%X{AD.requestGUID}] • We auto inject and correlate (one platform)
- 37. Please give us your feedback—Session T8455 • Complete the online survey you'll receive via email later today or via text at: Text this number: 878787 Text this word: APPSPHERE • Every time you submit a session survey, your name will be entered in a random drawing. We're giving away Amazon Echos to 5 lucky winners! • Thank you for your input APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 37 Win!
- 38. Thank you