Windows Debugging Tools - JavaOne 2013

Matthew Kilner – IBM Java L3 Service – Core team lead
23rd September 2013

Windows Debugging Tools

© 2013 IBM Corporation

Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE
USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS

2


About me
 Matthew Kilner


 Work for IBM
– 13 years working on IBM Java
• Memory Management
• Class Sharing
• RAS
– Currently leading the Core customer support team.


 Contact info
– kilnerm@uk.ibm.com
– Twitter: @IBMJTC
– Youtube: IBM_JTC


 Visit the IBM booth #5112 and meet other IBM developers
at JavaOne 2013
3


What should you get from this talk?

 Awareness of some excellent windows tools:
– A high level understanding of what it is they let you do,
– Details of where to obtain the tools,
– Details of where to find out how to use them.
–
 Some useful tips on working with the tools


General tips

 Most Microsoft tools require symbol files, often known as .pdb, to enable method names and source
lines in stack traces.
 Always keep your symbols in one place and keep a pre-configured Symbol path to hand.
• I will use generic paths for the JDK symbol files and overlay those paths with the correct level of
symbol for the JDK I am debugging at the time.
• JDK symbol files ship in the majority of the IBM JDK packaging formats.
 Remember to include the MS symbol server:
• SRV*<local symbol store>*http://msdl.microsoft.com/download/symbols

5


PDB files

 PDB files are an essential part of effective debugging

 The following link provides deailed information on how to generate them:
• http://msdn.microsoft.com/en-us/library/aa260783%28v=vs.60%29.aspx

 Requires the addition of compile and link time flags.

6


IBM Presentation Template Full Version

The tools

 Windows SDK tools:
– UMDH
– ADPlus
 Sysinternals tools:
– VMMAP
– Procdump
 Standard tools:
– Performance Monitor
 Stand alone tools:
– Debug Diagnostics

7


Usefulness is contextual

 A tools usefulness is limited to a number of problem types or tasks you are working on.

 Most of these tools have been “discovered” during analysis of real world problems.
– Predominantly related to Virtual addressing problems.
–
 Several of the tools have capabilities beyond the use described today.

8


Monitoring the system / application

 There are two common problem scenarios where it is desirable to understand more about how a
system or application is performing:
– Virtual Addressing problems
– Performance problems
• Slow responsiveness, High CPU usage, etc....



9


Performance Monitor
 Where to get it:
• Included in you windows installation
 What is it:
–A powerful monitoring tool
 What does it let us do:
–Monitor and analyse performance data in real time or from a log file.
–Can track almost every aspect of system performance.
 How do you use it:
–Launched by:
• Entering “perfmon” in your run dialog, or
• Via the Performance Monitor entry on the start menu
–Read the in-built help documentation.

10


Performance Monitor – monitoring process size

 The most common use is for
accurate measurement of a
processes size while debugging
virtual addressing issues.

 Define a data collector Set to record
the following counters on the process
object:
• Virtual bytes
• Private bytes

•

11


Performance Monitor – monitoring process size

 For live monitoring add the same
counters to the performance monitor
view

 When loading Data collection sets
back into the tool you will see similar
output.

•

12


Performance Monitor – other uses

 We primarily use it for monitoring process
sizes which only scratches the surface of
its practical uses.
 The ability to monitor nearly anything you
want on one or more processes such as:
• Monitoring per thread CPU
consumption across a process.
• Monitoring the power consumption of
individual CPU's.

13


Capturing Dumps

 Occasionally we see problems where the JDK does not write dumps when we expect it to.
– This provides a real challenge for continuing problem analysis.
–
 We use two tools that help capture dumps in this scenario:
– ADPlus (Windows SDK)
– Procmon (Sysinternals)
–


14


ADPlus

• http://msdn.microsoft.com/en-us/library/windows/desktop/hh852363.aspx
• Part of the Windows SDK
 What is it:
– A useful tool for generating system dumps
– Can also be used in hang situations
– Monitor a process and generate system dumps on:
• User request
• Process termination
• First and second chance exceptions.
– Documentation is included in the install directory
• <Path to installation>Windows Kits8.0Debuggersx64ADPlus.doc
15


Procdump

• http://technet.microsoft.com/en-us/sysinternals/dd996900
 What is it:
–A useful tool for generating system dumps
–Generate system dumps in a multitude of scenarios:
• The process exceeding a threshold on any performance counter
• On process exit
• On first and second chance exceptions.
–Command line utility
• See the link above for examples, or
• Run the tool with no options.
16


Procdump – usage examples

 Example: Write up to 3 dumps of a process named 'consume' when it exceeds 20% CPU usage for
five seconds to the directory c:dumpconsume with the name consume.dmp:
• C:>procdump -c 20 -s 5 -n 3 -o consume c:dumpconsume

 Example: Write 3 dumps 5 seconds apart:
• C:>procdump -s 5 -n 3 notepad.exe notepad.dmp

 Example: Launch a process and then monitor it for excessive CPU usage:
• C:>procdump -c 30 -s 10 -x consume.exe consume.dmp

17


Understanding Virtual Memory Growth

 A common problem we encounter is the virtual address space of processes growing and either:
– Causing process termination due to an inability to allocate memory, or
– Causing severe performance issues due to paging

 There are three excellent tools available on Windows to help understand what is causing the growth:
– UMDH (Windows SDK)
– Debug Diag ( Stand alone)
– VMMAP (Sysinternals)



18


UMDH

• http://msdn.microsoft.com/en-us/library/windows/desktop/hh852363.aspx
• Part of the Windows SDK
 What is it:
–A tool that tracks and analyzes windows heap allocations within a process
– Take snapshots of memory footprint within a process broken down by allocation stack.
– Compare snapshots and summarize the key differences.
– Identify stacks for specific allocation sizes.
– Basic usage instructions:
• http://msdn.microsoft.com/en-us/library/windows/hardware/ff558947%28v=vs.85%29.aspx
– More detail is available in this Microsoft knowledge base article:
• http://support.microsoft.com/kb/268343
–
19


UMDH – sample output
// Connecting to process 3312 ...
// Process 3312 opened handle=48.
// Loaded modules:

 Snapshot files contain the raw data
from a point of execution of the
process, you will see:
– A summary of loaded modules
– A summary fo the data format:
– A series of allocations:
–

 When you have multiple snapshots
you run the compare tool.

//

Base Size Module

//

13A0000

//

773B0000 1A9000 C:windowsSYSTEM32ntdll.dll

35000 C:Program FilesIBMCodeRallyjdkjrebinjavaw.exe

// Process modules enumerated.
// _NT_SYMBOL_PATH set by default to C:windowssymbols
// Debug library initialized ...
DBGHELP: ntdll - export symbols
*- - - - - - - - - - Start of data for heap @ 310000 - - - - - - - - - REQUESTED bytes + OVERHEAD at ADDRESS by BackTraceID
STACK if not already dumped.
*- - - - - - - - - - Heap 310000 Hogs - - - - - - - - - 28 bytes + 28 at 311310 by BackTrace720C0
7744BBED
773DC2E2
773F357E
773F1937
773DC34E

20


UMDH – comparing snapshots

 To run a comparison you execute:
• UMDH <log 1> <log 2> > <output file>

 Once the compare completes the output file contains a list of the call stacks that have differing memory
usage between the dumps sorted by overall size:
–

+ 550376 ( 550776 - 400) 12 allocsBackTrace33599F60
+
11 ( 12 1) BackTrace33599F60
allocations
ntdll!MD5Final+A63D
J9PRT26!j9mem_allocate_memory+74 (c:cygwinhomeforemansandboxjvm-binsrcj9portcommonj9memtag.c, 184)
j9vm26!allocateMemoryForSegment+A0 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 216)
j9vm26!allocateVirtualMemorySegmentInListInternal+E6 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 357)
j9vm26!allocateFixedMemorySegmentInList+A8 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 319)

21


UMDH – important considerations

 The tool uses snapshots of outstanding allocations so a degree of additional interpretation is needed.

 Do not assume the top consumer between two snapshots is the cause of the problem
• Also consider the volume of allocations, particularly for small chunks which have implications in
terms of fragmentation pre Vista/Server 2008.
•
 Best used in conjunction with performance monitor and over several iterations of snapshots.

 Longer durations between snapshots often make it easier to identify leaks

22


Debug Diagnostic Tool

• http://www.microsoft.com/en-us/download/details.aspx?id=26798
 What is it:
–A tool providing a range of capability for analysis of crash, hang or native memory isssues.
–Analysis of native memory issues.
–Basic analysis of crash or hang scenarios.
–Attaches to live processes and can be used to generate dumps at request or by rule configuration.
–Designed for minimal performance impact
– Basic Instructions:
• http://support.microsoft.com/kb/2580960
–Read the whitepaper:
• http://www.microsoft.com/download/en/details.aspx?id=23521
–Use the in-built help documentation.
23


Debug Diagnostic Tool – native memory analysis

 Our main use of this tool is for debuging native memory issues.
–The key thing to remember is to always inject the leak track dll, without this the reports you generate
will be basic.
–
–The optimal method of analysis involves taking multple dumps of the process as its virtual size
grows, understanding the rate of growth between dumps is important and can be monitored using
the performance monitor tool.
–
–Dumps can be generated directly from the UI or rules can be configured to generate dumps after a
set amount of time or increase in memory usage.
–
–Once dumps are generated you load them into the tool and select to run the Memory pressure
script. When complete the tool will open the report in a browser window.
–
24


Debug Diagnostic Tool – native memory report

 The report is broken into distinct sections:
• Virtual Memory Analysis Report
• Heap Analysis Report
• Leak Analysis Report (only with leak tracking)
•
 The following extracts are taken from results of monitoring a 64 bit process which has a maximum
address space of 8 Tbytes.
 Each report starts with a summary which contains some key pieces of information:


25


Debug Diagnostic Tool – Virtual Memory Analysis Report

 Contains:
• Virtual Memory Summary
• Virtual Memory Details
• Virtual Allocation Summary
• Loaded Module Summary
• Thread Summary
 The most pertinent details are the first
three elements
•

26


Debug Diagnostic Tool – Heap Analysis Report

 The heap analysis report
contains information on
the Heaps in use within
the process.

 Not to be confused with
the JVM's heap.


27


Debug Diagnostic Tool – Heap Details

 For each heap you get:
– A summary
– Information on the Heap segments
– Information on allocations
 The Summary:


28



 The Segment info:

29



 The Allocation Summary:

30


Debug Diagnostic Tool – Leak Analysis Report

 Only available when the leaktrack dll is injected to the process being monitored.
 The most useful aspect is the summary of allocations at the module level:
– By Count:
–


31


Debug Diagnostic Tool – Leak Analysis Report

– and by Size:
–


32


Debug Diagnostic Tool – Module Summary

 For each module we get a more detailed breakdown including a summary:


33



 We get a summary of functions by count and allocated size:


34



 And for each function additional detail:


35



 Allocation count and size breakdown:



36



 And sample call stacks for prominent sizes:



37


Debug Diagnostic Tool – Drawing conclusions

 The tool provides a lot of information, but doesn't do comparison of multiple data sets.
 Using a spreadsheet to capture the pertinent data points highlighted over the previous slides will build
a picture that allow you to see the key areas that are causing growth.
– You can even plot the data for easier visualization



38


VMMAP

• http://technet.microsoft.com/en-us/sysinternals/dd535533
 What is it:
–A tool for analysis the virtual and physical memory usage of a windows process
–Attaches to live processes for run time analysis and allows for export of process information in a
variety of formats.
–Provides a facility to launch a process and trace it which utilizes additional instrumentation that
enables us to identify the stacks allocating into specific regions of memory.
–Allows us to compare the contents of the address space between two different points in time, when
tracing is enabled you can see exactly who is allocating into regions.
–At any time you can flush the physical memory the process is using meaning this is a great tool for
seeing just how much memory a particular feature of your application uses.
–
–
39


VMMAP

 Having
attached or
launched a
process you
will see a
window like
this:
 Each time you
refresh the
view a new
snapshot is
created

40


VMMAP – Comparing snapshots

 First you have to open the timeline window:





 Next you left click and drag to select the
timeframe you wish to view:
• Snapshots are represented by the
small black lines:

41



 Once selected
you will see:

42



 Address ranges that are in the most recent snapshot but not in the earliest are highlighted in bright
green.
 Address ranges that were in the earliest snapshot but not in the current are highlighted in bright red.
 If you started the process under full tracing you can select a “HEAP” and view the allocations made
within it between the snapshots, for each allocation you can also view the call stack.



43


VMMAP – other features

 When tracing is enabled you have the option to view a trace of key memory operations by selecting the
trace option which lists:
• Allocation sites
• Number of allocations for the site
• The number of bytes allocated
• The type of operation
•
 You can also select any site and choose
to view its stack.

44


Q&A

 Questions

45


IBM@JavaOne

http://ibm.co/JavaOne2013

46


Windows Debugging Tools - JavaOne 2013

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Windows Debugging Tools - JavaOne 2013 (20)

Recently uploaded (20)

Windows Debugging Tools - JavaOne 2013