Windows Memory Dump Analysis

Accelerated
Version 6
Part 1: Process User Space

Dmitry Vostokov
Software Diagnostics Services
 WinDbg Commands
Prerequisites We use these boxes to
introduce WinDbg commands
used in practice exercises

Basic Windows troubleshooting

© 2023 Software Diagnostics Services

Training Goals
 Part 1A: Review fundamentals
 Part 1B: Review x64 disassembly
 Part 1C: Learn how to analyze process dumps
 Part 2A: Review fundamentals
 Part 2B: Review x64 disassembly
 Part 2C: Learn how to analyze kernel dumps
 Part 2D: Learn how to analyze complete
(physical memory) dumps
 Part 2E: Learn how to analyze minidumps

© 2023 Software Diagnostics Services

Training Principles
 Talk only about what I can show
 Lots of pictures
 Lots of examples
 Original content and examples

© 2023 Software Diagnostics Services

Coverage (Part 1)
 Windows 10 and 11
 Both x64* and x86 code, WOW64
 x64 disassembly review
 Preliminary .NET analysis
 Process memory dumps
 Crashes, hangs, memory and handle leaks,
CPU spikes
* Most of the exercises are focused on x64 code. For their x86 equivalents
from older Windows versions, please refer to the previous fourth edition of this
course.
© 2023 Software Diagnostics Services
Part 1A: Fundamentals

© 2023 Software Diagnostics Services

Process Space (x64)
00000000`00000000

User Space

00007FFF`FFFFFFFF
FFFF8000`00000000

Kernel Space

FFFFFFFF`FFFFFFFF

© 2023 Software Diagnostics Services

Process Space (x86)
00000000

User Space

7FFFFFFF
80000000

Kernel Space

FFFFFFFF

© 2023 Software Diagnostics Services

Application/Process/Module (x64)
00007FF6`00000000

Notepad Notepad.exe

user32.dll

User Space (PID 7212)

kernel32.dll
win32u

user32 win32u.dll
kernel32

ntdll ntdll.dll
00007FFF`FFFFFFFF
FFFF8000`00000000

Kernel Space

FFFFFFFF`FFFFFFFF

© 2023 Software Diagnostics Services

Application/Process/Module (x86)
00000000

Notepad Notepad.exe

user32.dll

User Space (PID 5772)

kernel32.dll
user32

kernel32 win32u.dll
win32u

ntdll ntdll.dll
7FFFFFFF
80000000

Kernel Space

FFFFFFFF

© 2023 Software Diagnostics Services

OS Kernel/Driver/Module (x64)
00000000`00000000

User Space

00007FFF`FFFFFFFF
FFFF8000`00000000

Driver.sys

Kernel Space

nt
Ntoskrnl.exe

Driver

FFFFFFFF`FFFFFFFF

© 2023 Software Diagnostics Services

OS Kernel/Driver/Module (x86)
00000000

User Space

7FFFFFFF
80000000

nt
Driver.sys

Kernel Space

Ntoskrnl.exe
Driver

FFFFFFFF

© 2023 Software Diagnostics Services

Process Virtual Space (x64)
00000000`00000000

00007FF6`00000000
Notepad

User Space (PID 7212)

win32u

user32

kernel32
00000000`00000000 ...
ntdll
00007FFF`FFFFFFFF FFFFFFFF`FFFFFFFF
FFFF8000`00000000

Kernel Space

nt

Driver

FFFFFFFF`FFFFFFFF

© 2023 Software Diagnostics Services

Process Virtual Space (x86)
00000000

Notepad

User Space (PID 5772)

user32

kernel32

win32u

ntdll 00000000 ... FFFFFFFF

7FFFFFFF
80000000

nt

Kernel Space

Driver

FFFFFFFF

© 2023 Software Diagnostics Services

Process Virtual Space (WOW64)
00000000`00000000
Notepad

kernel32
user32
win32u
ntdll_77b00000
00000000`FFFFFFFF
User Space (PID 9940)

00007FFF`DBE65000
wow64

ntdll
00007FFF`FFFFFFFF
FFFF8000`00000000

Kernel Space

nt

Driver

FFFFFFFF`FFFFFFFF

© 2023 Software Diagnostics Services

Process Memory Dump (x64)
00000000`00000000

00007FF6`00000000
Notepad
Notepad.dmp
User Space (PID 7212)
win32u

user32

kernel32

ntdll
00007FFF`FFFFFFFF
FFFF8000`00000000

WinDbg Commands
Kernel Space
lmv command lists modules
nt
and their description

Driver

FFFFFFFF`FFFFFFFF

© 2023 Software Diagnostics Services

Process Memory Dump (x86)
00000000

Notepad

User Space (PID 5772) Notepad.dmp

user32

kernel32

win32u

ntdll
7FFFFFFF
80000000

nt

WinDbg Commands
Kernel Space
lmv command lists modules
and their description
Driver

FFFFFFFF

© 2023 Software Diagnostics Services

Process Memory Dump (WOW64)
00000000`00000000
Notepad

kernel32
user32
win32u
ntdll_77b00000
00000000`FFFFFFFF Notepad.dmp
User Space (PID 9940)

00007FFF`DBE65000
wow64

ntdll
00007FFF`FFFFFFFF
FFFF8000`00000000

WinDbg Commands
Kernel Space
lmv command lists modules
nt
and their description

Driver

FFFFFFFF`FFFFFFFF

© 2023 Software Diagnostics Services

Process Threads
ApplicationA

User Space (PID 306) TID WinDbg Commands

TID 204
102
Process dumps:
user32 ~<n>s switches between
threads
ntdll

nt

Kernel Space

Driver

© 2023 Software Diagnostics Services

Thread Stack Raw Data
ApplicationA

TID TID
102 204 WinDbg Commands
User Space (PID 306)
User Stack for TID 204 Process dumps:
User Stack for TID 102 !teb

user32 Data:
dc / dps / dpp / dpa / dpu
ntdll

Kernel Stack for TID 204

nt

Kernel Space
Kernel Stack for TID 102

Driver

© 2023 Software Diagnostics Services

Thread Stack Trace
FunctionA()
User Stack for TID 102 {
...
FunctionB(); WinDbg Commands
...
}
FunctionB() 0:000> k
{
Return address Module!FunctionC+130 ...
Module!FunctionD
FunctionC(); Module!FunctionC+130
...
}
Module!FunctionB+220
Module!FunctionA+110
FunctionC()
{
Return address Module!FunctionB+220 ...
FunctionD();
...
}

Module!FunctionA

Resumes from address Saves return address

Module!FunctionA+110 Module!FunctionA+110

Module!FunctionB
Return address Module!FunctionA+110

Resumes from address Saves return address

Module!FunctionB+220 Module!FunctionB+220

Module!FunctionC

Resumes from address Saves return address

Module!FunctionC+130 Module!FunctionC+130

Module!FunctionD

© 2023 Software Diagnostics Services

Thread Stack Trace (no PDB)
FunctionA()
User Stack for TID 102 { Symbol file Module.pdb
...
FunctionB();
... FunctionA 22000 - 23000
} FunctionB 32000 - 33000
FunctionB() FunctionC 43000 – 44000
{ FunctionD 54000 - 55000
Return address Module+43130 ...
FunctionC();
...
}

FunctionC()
{
Return address Module+32220 ... No symbols for Module
FunctionD();
...
}

Module+22000

Resumes from address Saves return address

Module+22110 Module+22110
WinDbg Commands
Module+32000
Return address Module+22110
0:000> k
Resumes from address Saves return address Module+0
Module+32220 Module+32220
Module+43130
Module+32220
Module+43000
Module+22110
Resumes from address Saves return address
Module+43130 Module+43130

Module+54000

© 2023 Software Diagnostics Services

Exceptions (Access Violation)
M00000000
NULL pointer

ApplicationA

User Space (PID 306) WinDbg Commands

Minvalid memory access address=????????

ModuleA Set exception context

TID (process dump):
102 .cxr
TID
204

User Stack for TID 102

User Space (PID 306)

User Stack for TID 204

user32

ntdll

© 2023 Software Diagnostics Services

Exceptions (Runtime)
ApplicationA

User Space (PID 306)

M throws error ModuleA

TID
102
TID
204

User Stack for TID 102

User Space (PID 306)

User Stack for TID 204

user32

ntdll

© 2023 Software Diagnostics Services

Pattern-Oriented Diagnostic Analysis
Diagnostic Pattern: a common recurrent identifiable problem together with a set of
recommendations and possible solutions to apply in a specific context.

Diagnostic Problem: a set of indicators (symptoms, signs) describing a problem.

Diagnostic Analysis Pattern: a common recurrent analysis technique and method

of diagnostic pattern identification in a specific context.

Diagnostics Pattern Language: common names of diagnostic and diagnostic analysis

patterns. The same language for any operating system: Windows, Mac OS X, Linux, ...

Problem Resolution

Information Collection Information Extraction Problem Identification Troubleshooting

(Scripts) (Checklists) (Patterns) Suggestions

Debugging Strategy

Checklist: http://www.dumpanalysis.org/windows-memory-analysis-checklist

Patterns: http://www.dumpanalysis.org/blog/index.php/crash-dump-analysis-patterns/

© 2023 Software Diagnostics Services

Part 1B: x64 Disassembly

© 2023 Software Diagnostics Services

x64 CPU Registers
 RAX  EAX  AX  {AH, AL} RAX 64-bit EAX 32-bit

 ALU: RAX, RDX

 Counter: RCX

 Memory copy: RSI (src), RDI (dst)

 Stack: RSP

 Next instruction: RIP

 New: R8 – R15, Rx(D|W|B)

© 2023 Software Diagnostics Services

Instructions and Registers
 Opcode DST, SRC

 Examples:

mov rax, 10h ; RAX ← 0x10

mov r13, rdx ; R13 ← RDX
add r10, 10h ; R10 ← R10 + 0x10
imul edx, ecx ; EDX ← EDX * ECX
call rdx ; RDX already contains
; the address of func (&func)
; PUSH RIP; &func → RIP
sub rsp, 30h ; RSP ← RSP–0x30
; make room for local variables

© 2023 Software Diagnostics Services

Memory and Stack Addressing
Lower addresses Values

RSP-0x20 → [RSP-0x20]
RSP-0x18 → [RSP-0x18]
Stack grows

RSP-0x10 → [RSP-0x10]
RSP-0x8 → [RSP-0x8]
RSP → [RSP]
RSP+0x8 → [RSP+0x8]
RSP+0x10 → [RSP+0x10]
RSP+0x18 → [RSP+0x18]
RSP+0x20 → [RSP+0x20]

Higher addresses
© 2023 Software Diagnostics Services
Memory Cell Sizes
RSP → BYTE PTR [RSP]

RSP → DWORD PTR [RSP]

RSP → QWORD PTR [RSP]

RSP+0x8 →

RSP+0x8 →

RSP+0x8 →

© 2023 Software Diagnostics Services

Memory Load Instructions
 Opcode DST, PTR [SRC+Offset]

 Opcode DST

 Examples:

mov rax, qword ptr [rsp+10h] ; RAX ←

; 64-bit value at address RSP+0x10
mov ecx, dword ptr [20] ; ECX ←
; 32-bit value at address 0x20
pop rdi ; RDI ← value at address RSP
; RSP ← RSP + 8
lea r8, [rsp+20h] ; R8 ← address RSP+0x20

© 2023 Software Diagnostics Services

Memory Store Instructions
 Opcode PTR [DST+Offset], SRC

 Opcode DST|SRC

 Examples:

mov qword ptr [rbp-20h], rcx ; 64-bit value at address RBP-0x20

; ← RCX
mov byte ptr [0], 1 ; 8-bit value at address 0 ← 1
push rsi ; RSP ← RSP - 8
; value at address RSP ← RSI
inc dword ptr [rcx] ; 32-bit value at address RCX ←
; 1 + 32-bit value at address RCX

© 2023 Software Diagnostics Services

Flow Instructions
 Opcode DST

 Opcode PTR [DST]

 Examples:

jmp 00007ff6`9ef2f008 ; RIP ← 0x7ff69ef2f008

; (goto 0x7ff69ef2f008)
jmp qword ptr [rax+10h] ; RIP ← value at address RAX+0x10
call 00007ff6`9ef21400 ; RSP ← RSP – 8
00007ff6`9ef21057: ; value at address RSP ← 0x7ff69ef21057
; RIP ← 0x7ff69ef21400
; (goto 0x7ff69ef21400)

© 2023 Software Diagnostics Services

Windows API Parameters
 x86: Right to left PUSH

Args to Child are parameters

 x64: Left to right RCX, RDX, R8, R9, stack

Args to Child are not parameters

WinDbg Commands

0:000> kv
# Child-SP RetAddr : Args to Child : Call Site
…

© 2023 Software Diagnostics Services

Part 1C: Practice Exercises

© 2023 Software Diagnostics Services

Links
 Memory Dumps:

Included in Exercise 0

 Exercise Transcripts:

Included in this book

© 2023 Software Diagnostics Services

Exercise 0
 Goal: Install WinDbg or Debugging Tools for Windows, or pull
Docker image, and check that symbols are set up correctly

 Patterns: Stack Trace; Incorrect Stack Trace

 \AWMDA-Dumps\Exercise-0-Download-Setup-WinDbg.pdf

© 2023 Software Diagnostics Services

Process Memory Dumps

Exercises P1 – P21

© 2023 Software Diagnostics Services

Exercise P1
 Goal: Learn how to see dump file type and version, get a stack
trace, check its correctness, perform default analysis, list threads
and modules, check module version information, dump module
data, and check the process environment

 Patterns: Manual Dump (Process); Stack Trace; Not My Version

(Software); Environment Hint; Unknown Component

 \AWMDA-Dumps\Exercise-P1-Analysis-normal-process-dump-
wordpad-64.pdf

© 2023 Software Diagnostics Services

Exercise P2
 Goal: Repeat exercise P1 using 32-bit notepad process memory
dump

 \AWMDA-Dumps\Exercise-P2-Analysis-normal-process-dump-
wordpad-32.pdf

© 2023 Software Diagnostics Services

Exercise P3
 Goal: Learn how to list stack traces, check their correctness,
perform default analysis, list modules, check their version
information, and check thread age and CPU consumption

 Patterns: Stack Trace Collection (Unmanaged Space)

 \AWMDA-Dumps\Exercise-P3-Analysis-normal-process-dump-
msedge-64.pdf

© 2023 Software Diagnostics Services

Exercise P4
 Goal: Learn to recognize exceptions in process memory dumps
and get their context

 Patterns: Exception Stack Trace; Exception Module; Multiple

Exceptions (User Mode); NULL Pointer (Data)

 \AWMDA-Dumps\Exercise-P4-Analysis-process-dump-AppK-64-
no-symbols.pdf

© 2023 Software Diagnostics Services

Exercise P5
 Goal: Learn how to load application symbols

 \AWMDA-Dumps\Exercise-P5-Analysis-process-dump-AppK-64-
with-symbols.pdf

© 2023 Software Diagnostics Services

Exercise P6
 Goal: Learn how to recognize heap corruption, dump memory
contents, follow critical section wait chains, and check error and
status codes

 Patterns: Dynamic Memory Corruption (Process Heap); Wait

Chain (Critical Sections); Execution Residue (Unmanaged Space,
User); Last Error Collection

 \AWMDA-Dumps\Exercise-P6-Analysis-process-dump-AppL-
64.pdf

© 2023 Software Diagnostics Services

Exercise P7
 Goal: Learn how to debug heap corruption using page heap

 Patterns: Invalid Pointer (General); Instrumentation Information

 \AWMDA-Dumps\Exercise-P7-Analysis-process-dump-AppL2-
64.pdf

© 2023 Software Diagnostics Services

Exercise P8
 Goal: Learn how to recognize CPU spikes, invalid pointers,
disassemble code, and reconstruct stack trace

 Patterns: Wild Code; Active Thread; Spiking Thread; NULL

Pointer (Code); Truncated Stack Trace; Stored Exception

 \AWMDA-Dumps\Exercise-P8-Analysis-process-dump-AppM-
64.pdf

© 2023 Software Diagnostics Services

Exercise P9
 Goal: Learn how to recognize critical section waits and
deadlocks, dump raw stack data, and see hidden exceptions

 Patterns: Deadlock (Critical Sections); Hidden Exception (User

Space)

 \AWMDA-Dumps\Exercise-P9-Analysis-process-dump-AppN-
64.pdf

© 2023 Software Diagnostics Services

Deadlock
Thread 2
Thread 1 (waiting)
Critical Section
00007ff75e9b26d8

Thread 1
(owns)

Thread 2
(owns)

Thread 2
Critical Section
00007ff75e9b2700
Thread 1
(waiting)

© 2023 Software Diagnostics Services

Exercise P10
 Goal: Learn how to recognize application heap problems, buffer
and stack overflow patterns, and analyze raw stack data

 Patterns: Double Free (Process Heap); Local Buffer Overflow

(User Space); Stack Overflow (User Mode)

 \AWMDA-Dumps\Exercise-P10-Analysis-process-dump-AppO-
64.pdf

© 2023 Software Diagnostics Services

Exercise P11
 Goal: Learn how to analyze exception patterns, raw stacks, and
execution residue

 Patterns: Divide by Zero (User Mode); C++ Exception;

Execution Residue (Unmanaged Space, User)

 \AWMDA-Dumps\Exercise-P11-Analysis-process-dump-AppP-
64.pdf

© 2023 Software Diagnostics Services

Exercise P12
 Goal: Learn how to analyze managed space

 Patterns: Platform-Specific Debugger; CLR Thread; JIT Code

(.NET); Managed Code Exception; Managed Stack Trace

 \AWMDA-Dumps\Exercise-P12-Analysis-process-dump-AppR2-
64.pdf

© 2023 Software Diagnostics Services

Exercise P13
 Goal: Learn how to analyze the 32-bit process saved as a 64-bit
process memory dump

 Patterns: Virtualized Process (WOW64); Message Box;

Debugger Bug; Rough Stack Trace (Unmanaged Space)

 \AWMDA-Dumps\Exercise-P13-Analysis-process-dump-AppA-
WOW64.pdf

© 2023 Software Diagnostics Services

Exercise P14
 Goal: Learn how to analyze process memory leaks

 Patterns: Thread Age; Memory Leak (Process Heap)

 \AWMDA-Dumps\Exercise-P14-Analysis-process-dump-
AppS-64.pdf

© 2023 Software Diagnostics Services

Parameters and Locals

Debugging TV Frames episode 0x18

© 2023 Software Diagnostics Services

Symbol Types
 Exported and imported names

EXE DLL

 Function and variable names

 Data types

© 2023 Software Diagnostics Services

Exercise P15
 Goal: Learn how to navigate function parameters in cases of
reduced symbolic information in 32-bit process memory dumps

 Patterns: Reduced Symbolic Information

 \AWMDA-Dumps\Exercise-P15-Analysis-process-dump-notepad-
32.pdf

© 2023 Software Diagnostics Services

Exercise P16
 Goal: Learn how to navigate function parameters in x64 process
memory dumps

 Patterns: False Function Parameters; Injected Symbols

 \AWMDA-Dumps\Exercise-P16-Analysis-process-dump-notepad-
64.pdf

© 2023 Software Diagnostics Services

Exercise P17
 Goal: Learn how to navigate object wait chains in 32-bit memory
dumps saved with ProcDump

 Patterns: Embedded Comments; Wait Chain (General); No Data

Types; Deadlock (Mixed Objects, User Space)

 \AWMDA-Dumps\Exercise-P17-Analysis-process-dump-AppQ-
32.pdf

© 2023 Software Diagnostics Services

Exercise P18
 Goal: Learn how to navigate object wait chains in 64-bit memory
dumps saved with ProcDump

 Patterns: Not My Thread; Blocked Thread (Software); Main

Thread; Passive Thread (User Space); Coincidental Symbolic
Information

 \AWMDA-Dumps\Exercise-P18-Analysis-process-dump-AppQ-
64.pdf

© 2023 Software Diagnostics Services

Exercise P19
 Goal: Learn how to analyze process handle leaks

 Patterns: Active Space; Handle Leak

 \AWMDA-Dumps\Exercise-P19-Analysis-process-dump-AppT-
64.pdf

© 2023 Software Diagnostics Services

Exercise P20
 Goal: Learn how to analyze service memory dumps

 Patterns: Input Thread; Blocking Module

 \AWMDA-Dumps\Exercise-P20-Analysis-process-dump-ServiceA-
64.pdf

© 2023 Software Diagnostics Services

Exercise P21
 Goal: Learn how to analyze memory dumps from Rust processes

 Patterns: Language-Specific Subtrace (Rust)

 \AWMDA-Dumps\Exercise-P21-Analysis-process-dump-rusty.pdf

© 2023 Software Diagnostics Services

Pattern Links
Spiking Thread CLR Thread Active Space
C++ Exception Deadlock (Critical Sections) Debugger Bug
Divide by Zero (User Mode) Double Free (Process Heap) Exception Module
Dynamic Memory Corruption (Process Heap) Exception Stack Trace Blocking Module
Execution Residue (Unmanaged Space, User) Hidden Exception (User Space) Last Error Collection
Invalid Pointer (General) Local Buffer Overflow (User Space) Handle Leak
Manual Dump (Process) Managed Code Exception Input Thread
Managed Stack Trace Multiple Exceptions (User Mode)
Not My Version (Software) NULL Pointer (Data)
NULL Pointer (Code) Stack Trace
Stack Trace Collection (Unmanaged Space) Stack Overflow (User Mode)
Environment Hint Wild Code
Unknown Component Wait Chain (Critical Sections)
Virtualized Process (WOW64) Message Box
False Function Parameters Injected Symbols
Reduced Symbolic Information Truncated Stack Trace
Stored Exception Incorrect Stack Trace
Instrumentation Information Active Thread
JIT Code (.NET) Thread Age
Embedded Comment Wait Chain (General)
Deadlock (Mixed Object, User Space) Not My Thread
Blocked Thread (Software) Main Thread
Passive Thread (User Space) Coincidental Symbolic Information
Rough Stack Trace (Unmanaged Space) Platform-Specific Debugger
Memory Leak (Process Heap) Language-Specific Subtrace (Rust)

© 2023 Software Diagnostics Services

Pattern Classification

Space/Mode Memory dump type

Hooksware Wait Chain Patterns
DLL Link Patterns Insufficient Memory Patterns
Contention Patterns Stack Overflow Patterns
Stack Trace Patterns Symbol Patterns
Exception Patterns Meta-Memory Dump Patterns
Module Patterns Optimization Patterns
Thread Patterns Process Patterns
Dynamic Memory Corruption Patterns Deadlock and Livelock Patterns
.NET / CLR / Managed Space Patterns Executive Resource Patterns
Falsity and Coincidence Patterns RPC, LPC and ALPC Patterns
Hidden Artifact Patterns Pointer Patterns
Frame Patterns CPU Consumption Patterns

© 2023 Software Diagnostics Services

Pattern Case Studies
More than 70 multiple pattern case studies:

http://www.dumpanalysis.org/blog/index.php/pattern-cooperation/

Pattern Interaction chapters in

Memory Dump Analysis Anthology

© 2023 Software Diagnostics Services

Additional Resources
 WinDbg Help / WinDbg.org (quick links)
 DumpAnalysis.org / SoftwareDiagnostics.Institute / PatternDiagnostics.com
 Debugging.TV / YouTube.com/DebuggingTV / YouTube.com/PatternDiagnostics
 Windows Internals, 6th ed. (Chapter 14. Crash Dump Analysis), 7th ed.
 Advanced Windows Debugging
 Inside Windows Debugging
 Principles of Memory Dump Analysis
 Windows Debugging Notebook: Essential User Space WinDbg Commands
 Encyclopedia of Crash Dump Analysis Patterns, 3rd edition
 Memory Dump Analysis Anthology (Diagnomicon)

© 2023 Software Diagnostics Services

Further Training Courses
 Accelerated Windows Memory Dump Analysis, 6th Edition, Part 2

 Practical Foundations of Windows Debugging, Disassembling, Reversing, 2nd Edition

 Advanced Windows Memory Dump Analysis with Data Structures, 4th Edition, Revised

 Accelerated .NET Core Memory Dump Analysis, Revised Edition

 Accelerated Windows Malware Analysis with Memory Dumps, 3rd Edition

 Accelerated Disassembly, Reconstruction and Reversing, 2nd Revised Edition

 Accelerated Windows Debugging4, 3rd Edition

 Extended Windows Memory Dump Analysis

 Accelerated Windows API for Software Diagnostics

© 2023 Software Diagnostics Services

Q&A

Please send your feedback using the contact

form on PatternDiagnostics.com

© 2023 Software Diagnostics Services

Thank you for attendance!

© 2023 Software Diagnostics Services