Windows Memory

Dump Analysis

Dmitry Vostokov
Memory Dump Analysis Services
 WinDbg Commands

Prerequisites We use these boxes to

introduce WinDbg commands
used in practice exercises

Basic Windows troubleshooting

© 2011 Memory Dump Analysis Services

Training Goals
 Review fundamentals
 Learn how to analyze process dumps
 Learn how to analyze kernel dumps
 Learn how to analyze complete dumps

© 2011 Memory Dump Analysis Services

Training Principles
 Talk only about what I can show
 Lots of pictures
 Lots of examples
 Original content and examples

© 2011 Memory Dump Analysis Services

Schedule Summary
Day 1
 Analysis Fundamentals
 Process Memory Dumps

Day 2
 Kernel Memory Dumps
 Complete Memory Dumps

© 2011 Memory Dump Analysis Services

Part 1: Fundamentals

© 2011 Memory Dump Analysis Services

Memory/Kernel/User Space
FFFFFFFF

Kernel Space

80000000
7FFFFFFF

User Space

00000000

© 2011 Memory Dump Analysis Services

Application/Process/Module
FFFFFFFF

Kernel Space

80000000
7FFFFFFF
user32

user32.dll

User Space (PID 102)

Notepad.exe
Notepad

00000000

© 2011 Memory Dump Analysis Services

OS Kernel/Driver/Module
FFFFFFFF

Driver
Driver.sys

Kernel Space

Ntoskrnl.exe
nt

80000000
7FFFFFFF

User Space

00000000

© 2011 Memory Dump Analysis Services

Process Virtual Space
FFFFFFFF

Driver

Kernel Space

00000000 ... FFFFFFFF

nt

80000000
7FFFFFFF
user32

User Space (PID 102)

Notepad

00000000

© 2011 Memory Dump Analysis Services

Process Memory Dump
FFFFFFFF

Driver

WinDbg Commands
Kernel Space lmv command lists modules
and their description

nt

80000000
7FFFFFFF
user32

User Space (PID 102) Notepad.exe.102.dmp

Notepad

00000000

© 2011 Memory Dump Analysis Services

Kernel Memory Dump
FFFFFFFF

Driver

MEMORY.DMP
Kernel Space

nt

80000000
7FFFFFFF
user32

WinDbg Commands

lmv command lists modules

and their description
User Space (PID 102)

Notepad

00000000

© 2011 Memory Dump Analysis Services

Complete Memory Dump
FFFFFFFF

WinDbg Commands
Driver
.process switches between
process virtual spaces (kernel
space part remains the same)
Kernel Space

nt

80000000
7FFFFFFF MEMORY.DMP

user32 user32

User Space User Space

(PID 102) (PID 204)

Notepad
Calc

00000000

© 2011 Memory Dump Analysis Services

Process Threads
Driver
TID
102
WinDbg Commands
Kernel Space Process dumps:
~<n>s switches between
threads
nt TID
204 Kernel/Complete dumps:
~<n>s switches between
processors
.thread switches between
ntdll
threads
user32

User Space (PID 306)

ApplicationA

© 2011 Memory Dump Analysis Services

System Threads
Driver

WinDbg Commands
TID
306 Kernel Space Kernel/Complete dumps:
~<n>s switches between
processors
.thread switches between
nt
threads

ntdll

user32

User Space (PID 306)

ApplicationA

© 2011 Memory Dump Analysis Services

Thread Stack Raw Data
Driver
TID
102
WinDbg Commands
Kernel Stack for TID 102

Kernel Space Process dumps:

Kernel Stack for TID 204
!teb

nt Kernel dumps:
TID
204 !thread

Complete dumps:
!teb for user space
ntdll
!thread for kernel space
user32
Data:
dc / dps / dpp / dpa / dpu
User Stack for TID 102

User Space (PID 306)

User Stack for TID 204

ApplicationA

© 2011 Memory Dump Analysis Services

Thread Stack Trace
FunctionA()
User Stack for TID 102 {
...
FunctionB();

}
...
WinDbg Commands
FunctionB()
{
Return address Module!FunctionC+130 ... 0:000> k
FunctionC(); Module!FunctionD
...
} Module!FunctionC+130
FunctionC()
Module!FunctionB+220
{ Module!FunctionA+110
Return address Module!FunctionB+220 ...
FunctionD();
...
}

Module!FunctionA

Resumes from address Saves return address

Module!FunctionA+110 Module!FunctionA+110

Module!FunctionB
Return address Module!FunctionA+110

Resumes from address Saves return address

Module!FunctionB+220 Module!FunctionB+220

Module!FunctionC

Resumes from address Saves return address

Module!FunctionC+130 Module!FunctionC+130

Module!FunctionD

© 2011 Memory Dump Analysis Services

Thread Stack Trace (no PDB)
FunctionA()
User Stack for TID 102 { Symbol file Module.pdb
...
FunctionB();
... FunctionA 22000 - 23000
} FunctionB 32000 - 33000
FunctionB() FunctionC 43000 – 44000
{ FunctionD 54000 - 55000
Return address Module+43130 ...
FunctionC();
...
}

FunctionC()
{
Return address Module+32220 ... No symbols for Module
FunctionD();
...
} WinDbg Commands
Module+22000
0:000> k
Resumes from address Saves return address Module+0
Module+22110 Module+22110 Module+43130
Module+32220
Module+32000
Return address Module+22110 Module+22110
Resumes from address Saves return address
Module+32220 Module+32220

Module+43000

Resumes from address Saves return address

Module+43130 Module+43130

Module+54000

© 2011 Memory Dump Analysis Services

Exceptions (Access Violation)
ntdll

User Space (PID 306)

user32
WinDbg Commands
ModuleA
address=????????
Minvalid memory access
Set exception context
(process dump):
TID TID .cxr
102 204
Set trap context
(kernel/complete dump):
.trap
User Stack for TID 102 Check address:
!pte
User Space (PID 306)
User Stack for TID 204

ApplicationA

NULL pointer
M00000000

© 2011 Memory Dump Analysis Services

Exceptions (Runtime)
ntdll

User Space (PID 306)

user32

M throws error ModuleA

TID TID
102 204

User Stack for TID 102

User Space (PID 306)

User Stack for TID 204

ApplicationA

© 2011 Memory Dump Analysis Services

Pattern-Driven Analysis

Pattern: a common recurrent identifiable problem together with a set of

recommendations and possible solutions to apply in a specific context

Problem Resolution

Information Collection Information Extraction Problem Identification Troubleshooting

(Scripts) (Checklists) (Patterns) Suggestions

Debugging Strategy

Checklist: http://www.dumpanalysis.org/blog/index.php/2007/06/20/crash-
dump-analysis-checklist/
Patterns: http://www.dumpanalysis.org/blog/index.php/crash-dump-
analysis-patterns/

© 2011 Memory Dump Analysis Services

Part 2: Practice Exercises

© 2011 Memory Dump Analysis Services

Links
 Memory Dumps:
Removed from public preview version

 Exercise Transcripts:
Removed from public preview version

© 2011 Memory Dump Analysis Services

Exercise 0
 Goal: Install Debugging Tools for Windows and learn how to
set up symbols correctly

 Patterns: Incorrect Stack Trace

© 2011 Memory Dump Analysis Services

Process Memory Dumps

Exercises P1-P13

© 2011 Memory Dump Analysis Services

Exercise P1
 Goal: Learn how to see dump file type and version, get a
stack trace, check its correctness, perform default analysis,
list modules, check their version information, check process
environment

 Patterns: Manual Dump; Stack Trace; Not My Version;

Environment Hint

© 2011 Memory Dump Analysis Services

Exercise P2
 Goal: Learn how to list stack traces, check their correctness,
perform default analysis, list modules, check their version
information, check process environment; dump module data

 Patterns: Manual Dump; Stack Trace; Not My Version;

Environment Hint; Unknown Component

© 2011 Memory Dump Analysis Services

Exercise P3
 Goal: Learn how to list stack traces, check their correctness,
perform default analysis, list modules, check their version
information, check thread age and CPU consumption

 Patterns: Stack Trace Collection

© 2011 Memory Dump Analysis Services

Exercise P4
 Goal: Learn to recognize exceptions in process memory
dumps and get their context

 Patterns: Exception Thread; Multiple Exceptions; NULL

Pointer

© 2011 Memory Dump Analysis Services

Exercise P5
 Goal: Learn how to load application symbols, recognize
exceptions in process memory dumps and get their context

 Patterns: Exception Thread; Multiple Exceptions; NULL

Pointer

© 2011 Memory Dump Analysis Services

Exercise P6
 Goal: Learn how to recognize heap corruption

 Patterns: Exception Thread; Dynamic Memory Corruption

© 2011 Memory Dump Analysis Services

Exercise P7
 Goal: Learn how to recognize heap corruption and check
error and status codes

 Patterns: Exception Thread; Dynamic Memory Corruption

© 2011 Memory Dump Analysis Services

Exercise P8
 Goal: Learn how to recognize CPU spikes, invalid pointers
and disassemble code

 Patterns: Exception Thread; Wild Code; CPU Spike; Mutiple

Exceptions; NULL Code Pointer; Invalid Pointer

© 2011 Memory Dump Analysis Services

Exercise P9
 Goal: Learn how to recognize critical section waits and
deadlocks, dump raw stack data and see hidden exceptions

 Patterns: Wait Chain; Deadlock; Hidden Exception

© 2011 Memory Dump Analysis Services

Deadlock
Thread 2
Thread 1 (waiting)
Critical Section
000000013fd7ef08

Thread 1
(owns)

Thread 2
(owns)

Thread 2
Critical Section
000000013fd7eee0
Thread 1
(waiting)

© 2011 Memory Dump Analysis Services

Exercise P10
 Goal: Learn how to recognize application heap problems,
buffer and stack overflow patterns and analyze raw stack
data

 Patterns: Double Free; Local Buffer Overflow; Stack

Overflow

© 2011 Memory Dump Analysis Services

Exercise P11
 Goal: Learn how to analyze various patterns, raw stacks and
execution residue

 Patterns: Divide by Zero; C++ Exception; Multiple

Exceptions; Execution Residue

© 2011 Memory Dump Analysis Services

Exercise P12
 Goal: Learn how to load the correct .NET WinDbg extension
and analyze managed space

 Patterns: CLR Thread; Version-Specific Extension;

Managed Code Exception; Managed Stack Trace

© 2011 Memory Dump Analysis Services

Exercise P13
 Goal: Learn how to analyze 32-process saved as a 64-bit
process memory dump

 Patterns: Virtualized Process; Message Box; Execution

Residue

© 2011 Memory Dump Analysis Services

Exercise P14
 Goal: Learn how process memory leaks

 Patterns: Spiking Thread; Thread Age; Memory Leak

(process heap)

© 2011 Memory Dump Analysis Services

Pattern Links
Spiking Thread CLR Thread
C++ Exception Critical Section Deadlock
Divide by Zero Double Free
Heap Corruption Exception Stack Trace
Execution Residue Hidden Exception
Invalid Pointer Local Buffer Overflow
Manual Dump Managed Code Exception
Managed Stack Trace Multiple Exceptions
Not My Version NULL Data Pointer
NULL Code Pointer Stack Trace
Stack Trace Collection Stack Overflow
Environment Hint Wild Code
Unknown Component Wait Chain
Virtualized Process Message Box
Version-Specific Extension Memory Leak

© 2011 Memory Dump Analysis Services

Kernel Memory Dumps

Exercises K1-K5

© 2011 Memory Dump Analysis Services

Exercise K1
 Goal: Learn how to get various information related to
hardware, system, sessions, processes, threads and
modules

 Patterns: Invalid Pointer; Virtualized System; Stack Trace

Collection

© 2011 Memory Dump Analysis Services

Exercise K2
 Goal: Learn how to check and compare kernel pool usage

 Patterns: Manual Dump; Insufficient Memory (kernel pool)

© 2011 Memory Dump Analysis Services

Exercise K3
 Goal: Learn how to recognize pool corruption and check
pool data

 Patterns: Dynamic Memory Corruption (kernel pool);

Execution Residue

© 2011 Memory Dump Analysis Services

Exercise K4
 Goal: Learn how to check hooked or invalid code and kernel
raw stack

 Patterns: Null Pointer; Hooked Functions (kernel pool);

Execution Residue; Coincidental Symbolic Information

© 2011 Memory Dump Analysis Services

Exercise K5
 Goal: Learn how to check I/O requests

 Patterns: Blocking File

© 2011 Memory Dump Analysis Services

Pattern Links
Manual Dump Invalid Pointer
Virtualized System Stack Trace Collection
Insufficient Memory Dynamic Memory Corruption
Execution Residue Null Pointer
Hooked Functions Coincidental Symbolic Information
Blocking File

© 2011 Memory Dump Analysis Services

Complete Memory Dumps

Exercises C1-C2

© 2011 Memory Dump Analysis Services

Memory Spaces
 Complete memory == Physical memory
 We always see the current process space

WinDbg Commands
Context switch switching to a different process
context:

.process /r /p

© 2011 Memory Dump Analysis Services

Major Challenges
 Multiple processes (user spaces) to examine
 User space view needs to be correct when we examine another thread

WinDbg Commands
dump all stack traces:

!process 0 ff

User Space

© 2011 Memory Dump Analysis Services

Common Commands
 .logopen <file>
Opens a log file to save all subsequent output

 View commands
Dump everything or selected processes and threads (context changes automatically)

 Switch commands
Switch to a specific process or thread for a fine-grain analysis

© 2011 Memory Dump Analysis Services

View Commands
 !process 0 ff
Lists all processes (including times, environment, modules) and their thread stack traces

 !process 0 1f
The same as the previous command but without PEB information (more secure)

 !process <address> ff or !process <address> 1f

The same as the previous commands but only for an individual process

 !thread <address> ff
Shows thread information and stack trace

 !thread <address> f6
The same as the previous command but shows the first 3 parameters for every function

© 2011 Memory Dump Analysis Services

Switch Commands
 .process /r /p <address>
Switches to a specified process. Its context becomes current. Reloads symbol files for user space.
Now we can use commands like !cs

0: kd> .process /r /p fffffa80044d8b30

Implicit process is now fffffa80`044d8b30
Loading User Symbols
.................................

 .thread <address>
Switches to a specified thread. Assumes the current process context
Now we can use commands like k*

 .thread /r /p <address>
The same as the previous command but makes the thread process context current and reloads
symbol files for user space:

0: kd> .thread /r /p fffffa80051b7060

Implicit thread is now fffffa80`051b7060
Implicit process is now fffffa80`044d8b30
Loading User Symbols
.................................

© 2011 Memory Dump Analysis Services

Exercise C1
 Goal: Learn how to get various information related to
processes, threads and modules

 Patterns: Stack Trace Collection

© 2011 Memory Dump Analysis Services

Example: Blocked Thread
THREAD fffffa800451db60 Cid 07f4.0b8c Teb: 000007fffffd6000 Win32Thread: fffff900c27c0c30 WAIT: (WrUserRequest) UserMode Non-
Alertable
fffffa8004e501e0 SynchronizationEvent
Not impersonating
DeviceMap fffff8a001e84c00
Owning Process fffffa8004514630 Image: ApplicationA.exe
[...]
Stack Init fffff88005b7fdb0 Current fffff88005b7f870
Base fffff88005b80000 Limit fffff88005b77000 Call 0
Priority 11 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffff880`05b7f8b0 fffff800`01a93992 nt!KiSwapContext+0x7a
fffff880`05b7f9f0 fffff800`01a95cff nt!KiCommitThreadWait+0x1d2
fffff880`05b7fa80 fffff960`0011b557 nt!KeWaitForSingleObject+0x19f
fffff880`05b7fb20 fffff960`0011b5f1 win32k!xxxRealSleepThread+0x257
fffff880`05b7fbc0 fffff960`0012e22e win32k!xxxSleepThread+0x59
fffff880`05b7fbf0 fffff800`01a8b993 win32k!NtUserWaitMessage+0x46
fffff880`05b7fc20 00000000`775cbf5a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`05b7fc20)
00000000`022ff7c8 00000000`775d7214 USER32!ZwUserWaitMessage+0xa
00000000`022ff7d0 00000000`775d74a5 USER32!DialogBox2+0x274
00000000`022ff860 00000000`776227f0 USER32!InternalDialogBox+0x135
00000000`022ff8c0 00000000`77621ae5 USER32!SoftModalMessageBox+0x9b4
00000000`022ff9f0 00000000`7762133b USER32!MessageBoxWorker+0x31d
00000000`022ffbb0 00000000`77621232 USER32!MessageBoxTimeoutW+0xb3
>>> 00000000`022ffc80 00000001`3f3c1089 USER32!MessageBoxW+0x4e
00000000`022ffcc0 00000001`3f3c11fb ApplicationA+0x1089
00000000`022ffcf0 00000001`3f3c12a5 ApplicationA+0x11fb
00000000`022ffd20 00000000`776cf56d ApplicationA+0x12a5
00000000`022ffd50 00000000`77803281 kernel32!BaseThreadInitThunk+0xd
00000000`022ffd80 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

© 2011 Memory Dump Analysis Services

Example: Wait Chain
THREAD fffffa8004562b60 Cid 0b34.0858 Teb: 000007fffffae000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-
Alertable
>>> fffffa8004b96ce0 Mutant - owning thread fffffa8004523b60
Not impersonating
DeviceMap fffff8a001e84c00
Owning Process fffffa8005400b30 Image: ApplicationC.exe
Attached Process N/A Image: N/A
Wait Start TickCount 36004 Ticks: 4286 (0:00:01:06.862)
Context Switch Count 2
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address ApplicationC (0x000000013f7012a0)
Stack Init fffff88005b1ddb0 Current fffff88005b1d900
Base fffff88005b1e000 Limit fffff88005b18000 Call 0
Priority 11 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5
Child-SP RetAddr Call Site
fffff880`05b1d940 fffff800`01a93992 nt!KiSwapContext+0x7a
fffff880`05b1da80 fffff800`01a95cff nt!KiCommitThreadWait+0x1d2
fffff880`05b1db10 fffff800`01d871d2 nt!KeWaitForSingleObject+0x19f
fffff880`05b1dbb0 fffff800`01a8b993 nt!NtWaitForSingleObject+0xb2
fffff880`05b1dc20 00000000`7781fefa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`05b1dc20)
00000000`00e2f658 000007fe`fda910ac ntdll!NtWaitForSingleObject+0xa
00000000`00e2f660 00000001`3f70112e KERNELBASE!WaitForSingleObjectEx+0x79
00000000`00e2f700 00000001`3f70128b ApplicationC+0x112e
00000000`00e2f730 00000001`3f701335 ApplicationC+0x128b
00000000`00e2f760 00000000`776cf56d ApplicationC+0x1335
00000000`00e2f790 00000000`77803281 kernel32!BaseThreadInitThunk+0xd
00000000`00e2f7c0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

© 2011 Memory Dump Analysis Services

Example: Consumption
1: kd> !vm 1: kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
*** Virtual Memory Usage *** PROCESS fffffa8003baa890
Physical Memory: 1031581 ( 4126324 Kb) SessionId: none Cid: 0004 Peb: 00000000 ParentCid: 0000
Page File: \??\C:\pagefile.sys DirBase: 00187000 ObjectTable: fffff8a000001a80 HandleCount: 558.
Current: 4433524 Kb Free Space: 4433520 Kb Image: System
Minimum: 4433524 Kb Maximum: 12378972 Kb
Available Pages: 817652 ( 3270608 Kb) PROCESS fffffa8004277870
ResAvail Pages: 965229 ( 3860916 Kb) SessionId: none Cid: 011c Peb: 7fffffdf000 ParentCid: 0004
Locked IO Pages: 0 ( 0 Kb) DirBase: 133579000 ObjectTable: fffff8a00000f3d0 HandleCount: 35.
Free System PTEs: 33555714 ( 134222856 Kb) Image: smss.exe
Modified Pages: 15794 ( 63176 Kb)
Modified PF Pages: 15793 ( 63172 Kb) PROCESS fffffa80048f3950
NonPagedPool Usage: 88079121 ( 352316484 Kb) SessionId: 0 Cid: 016c Peb: 7fffffdf000 ParentCid: 0154
NonPagedPoolNx Usage: 12885 ( 51540 Kb) DirBase: 128628000 ObjectTable: fffff8a001d62f90 HandleCount: 387.
NonPagedPool Max: 764094 ( 3056376 Kb) Image: csrss.exe
>>> ********** Excessive NonPaged Pool Usage *****
PagedPool 0 Usage: 35435 ( 141740 Kb)
PagedPool 1 Usage: 3620 ( 14480 Kb) [...]
PagedPool 2 Usage: 573 ( 2292 Kb)
PagedPool 3 Usage: 535 ( 2140 Kb) PROCESS fffffa800541a060
PagedPool 4 Usage: 538 ( 2152 Kb) SessionId: 1 Cid: 0b94 Peb: 7fffffde000 ParentCid: 06ac
PagedPool Usage: 40701 ( 162804 Kb) >>> DirBase: a6ba9000 ObjectTable: fffff8a0098efaf0 HandleCount:
PagedPool Maximum: 33554432 ( 134217728 Kb) 20013.
Session Commit: 9309 ( 37236 Kb) Image: ApplicationE.exe
Shared Commit: 6460 ( 25840 Kb)
Special Pool: 0 ( 0 Kb) [...]
Shared Process: 5760 ( 23040 Kb)
PagedPool Commit: 40765 ( 163060 Kb)
Driver Commit: 2805 ( 11220 Kb)
Committed pages: 212472 ( 849888 Kb)
Commit limit: 2139487 ( 8557948 Kb)

© 2011 Memory Dump Analysis Services

Example: Corruption
THREAD fffffa8004514060 Cid 0abc.087c Teb: 000007fffffae000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode
Alertable
fffffa800518fb30 ProcessObject
[...]
Child-SP RetAddr Call Site
fffff880`05a6c940 fffff800`01a93992 nt!KiSwapContext+0x7a
fffff880`05a6ca80 fffff800`01a95cff nt!KiCommitThreadWait+0x1d2
fffff880`05a6cb10 fffff800`01d871d2 nt!KeWaitForSingleObject+0x19f
fffff880`05a6cbb0 fffff800`01a8b993 nt!NtWaitForSingleObject+0xb2
fffff880`05a6cc20 00000000`7781fefa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`05a6cc20)
00000000`00dde928 00000000`77895ce2 ntdll!NtWaitForSingleObject+0xa
00000000`00dde930 00000000`77895e85 ntdll!RtlReportExceptionEx+0x1d2
00000000`00ddea20 00000000`77895eea ntdll!RtlReportException+0xb5
00000000`00ddeaa0 00000000`77896d25 ntdll!RtlpTerminateFailureFilter+0x1a
00000000`00ddead0 00000000`777e5148 ntdll!RtlReportCriticalFailure+0x96
00000000`00ddeb00 00000000`7780554d ntdll!_C_specific_handler+0x8c
00000000`00ddeb70 00000000`777e5d1c ntdll!RtlpExecuteHandlerForException+0xd
00000000`00ddeba0 00000000`777e62ee ntdll!RtlDispatchException+0x3cb
00000000`00ddf280 00000000`77896cd2 ntdll!RtlRaiseException+0x221
00000000`00ddf8c0 00000000`77897396 ntdll!RtlReportCriticalFailure+0x62
00000000`00ddf990 00000000`778986c2 ntdll!RtlpReportHeapFailure+0x26
00000000`00ddf9c0 00000000`7789a0c4 ntdll!RtlpHeapHandleError+0x12
00000000`00ddf9f0 00000000`7783d1cd ntdll!RtlpLogHeapFailure+0xa4
00000000`00ddfa20 00000000`776d2c7a ntdll! ?? ::FNODOBFM::`string'+0x123b4
>>> 00000000`00ddfaa0 00000001`3fa71274 kernel32!HeapFree+0xa
00000000`00ddfad0 00000001`3fa710c3 ApplicationD+0x1274
00000000`00ddfb00 00000001`3fa71303 ApplicationD+0x10c3
00000000`00ddfb30 00000001`3fa713ad ApplicationD+0x1303
00000000`00ddfb60 00000000`776cf56d ApplicationD+0x13ad
00000000`00ddfb90 00000000`77803281 kernel32!BaseThreadInitThunk+0xd
00000000`00ddfbc0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

© 2011 Memory Dump Analysis Services

Example: Special Process
1: kd> !vm

[...]

0744 svchost.exe 19725 ( 78900 Kb)

06ac explorer.exe 11444 ( 45776 Kb)
0920 iexplore.exe 8828 ( 35312 Kb)
0354 svchost.exe 5589 ( 22356 Kb)
040c audiodg.exe 4003 ( 16012 Kb)
0334 svchost.exe 3852 ( 15408 Kb)
04e4 spoolsv.exe 3230 ( 12920 Kb)
012c svchost.exe 2802 ( 11208 Kb)
0168 iexplore.exe 2106 ( 8424 Kb)
0384 svchost.exe 2090 ( 8360 Kb)
042c svchost.exe 1938 ( 7752 Kb)
0218 lsass.exe 1314 ( 5256 Kb)
03d4 svchost.exe 1128 ( 4512 Kb)
>>> 0a78 WerFault.exe 1107 ( 4428 Kb)
0210 services.exe 1106 ( 4424 Kb)
0288 svchost.exe 980 ( 3920 Kb)
02d8 svchost.exe 891 ( 3564 Kb)
0438 msdtc.exe 851 ( 3404 Kb)
071c mscorsvw.exe 821 ( 3284 Kb)
0378 taskhost.exe 795 ( 3180 Kb)
01a8 psxss.exe 685 ( 2740 Kb)
08a0 jusched.exe 667 ( 2668 Kb)
09e0 jucheck.exe 621 ( 2484 Kb)
0828 mscorsvw.exe 600 ( 2400 Kb)
0538 mdm.exe 595 ( 2380 Kb)
0220 lsm.exe 595 ( 2380 Kb)
[...]

© 2011 Memory Dump Analysis Services

Exercise C2
 Goal: Learn how to recognize various abnormal software
behavior patterns

 Patterns: Special Process; Handle Leak; Spiking Thread;

Stack Trace Collection; Message Box; Wait Chain; Exception
Thread

© 2011 Memory Dump Analysis Services

Wait Chain
process process
Thread ApplicationC ApplicationB
Thread 886ee030
83336a00 (waiting)
Critical Section
00a9b7c0

Thread Thread
830f9990 832be6d8
Thread (waiting)
83336a00
(owns)
Mutant
00a9b7c0
Thread
886ee030
(owns)

Thread
Thread 83336a00
886ee030 (waiting)
Critical Section
00a9b7a8
Thread
832be6d8
(owns)

© 2011 Memory Dump Analysis Services

Pattern Links
Special Process
Handle Leak
Spiking Thread
Stack Trace Collection
Message Box
Wait Chain (critical sections)
Exception Stack Trace
Wait Chain (window messaging)
Wait Chain (LPC/ALPC)

© 2011 Memory Dump Analysis Services

Common Mistakes
 Not switching to the appropriate context
 Not looking at full stack traces
 Not looking at all stack traces
 Not using checklists
 Not looking past the first found evidence

Note: Listing both x86 and x64 stack traces

http://www.dumpanalysis.org/blog/index.php/2010/02/09/complete-stack-traces-from-x64-system/

© 2011 Memory Dump Analysis Services

Kernel Minidumps

Memory Dump Analysis Anthology, Volume 1

pp. 43 - 67

© 2011 Memory Dump Analysis Services

Pattern Case Studies
60 multiple pattern case studies:

http://www.dumpanalysis.org/blog/index.php/
pattern-cooperation/

Pattern Interaction chapters in

Memory Dump Analysis Anthology

© 2011 Memory Dump Analysis Services

Resources
 WinDbg Help / WinDbg.org (quick links)
 DumpAnalysis.org
 Windows Internals, 5th ed.
 Windows Debugging: Practical Foundations
 x64 Windows Debugging: Practical Foundations
 Advanced Windows Debugging
 Windows Debugging Notebook: Essential User Space WinDbg Commands
 Memory Dump Analysis Anthology

© 2011 Memory Dump Analysis Services

Q&A

Please send your feedback using the contact

form on DumpAnalysis.com

© 2011 Memory Dump Analysis Services

Thank you for attendance!

© 2011 Memory Dump Analysis Services