(Ebook) Accelerated macOS Core Dump Analysis: Training Course Transcript with LLDB Practice Exercises by Dmitry Vostokov, Software Diagnostics Services ISBN 9781912636754, 1912636751 All Chapters Instant Download
(Ebook) Accelerated macOS Core Dump Analysis: Training Course Transcript with LLDB Practice Exercises by Dmitry Vostokov, Software Diagnostics Services ISBN 9781912636754, 1912636751 All Chapters Instant Download
com
OR CLICK HERE
DOWLOAD EBOOK
https://ebooknice.com/product/accelerated-linux-core-dump-analysis-
third-edition-47839386
ebooknice.com
https://ebooknice.com/product/memory-dump-analysis-anthology-
volume-2-revised-edition-53663124
ebooknice.com
ebooknice.com
(Ebook) Memory Dump Analysis Anthology, Volume 3, Revised
Edition by Dmitry Vostokov, Software Diagnostics Institute
https://ebooknice.com/product/memory-dump-analysis-anthology-
volume-3-revised-edition-53663130
ebooknice.com
https://ebooknice.com/product/memory-dump-analysis-anthology-
volume-2-revised-edition-53663118
ebooknice.com
ebooknice.com
ebooknice.com
ebooknice.com
Published by OpenTask, Republic of Ireland
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted, in any form or by any means, without the publisher’s prior written permission.
Product and company names mentioned in this book may be trademarks of their owners.
OpenTask books and magazines are available through booksellers and distributors worldwide.
For further information or comments, send requests to [email protected].
A CIP catalog record for this book is available from the British Library.
2
Contents
3
4
About the Author
5
Dmitry Vostokov is an internationally recognized expert, speaker,
educator, scientist, inventor, and author. He is the founder of the
pattern-oriented software diagnostics, forensics, and prognostics
discipline (Systematic Software Diagnostics), and Software Diagnostics
Institute (DA+TA: DumpAnalysis.org + TraceAnalysis.org). Vostokov has
also authored more than 50 books on software diagnostics, anomaly
detection and analysis, software and memory forensics, root cause
analysis and problem solving, memory dump analysis, debugging,
software trace and log analysis, reverse engineering, and malware
analysis. He has over 25 years of experience in software architecture, design, development, and
maintenance in various industries, including leadership, technical, and people management roles.
Dmitry also founded Syndromatix, Anolog.io, BriteTrace, DiaThings, Logtellect, OpenTask Iterative
and Incremental Publishing (OpenTask.com), Software Diagnostics Technology and Services (former
Memory Dump Analysis Services) PatternDiagnostics.com, and Software Prognostics. In his spare
time, he presents various topics on Debugging.TV and explores Software Narratology, its further
development as Narratology of Things and Diagnostics of Things (DoT), Software Pathology, and
Quantum Software Diagnostics. His current interest areas are theoretical software diagnostics and its
mathematical and computer science foundations, application of formal logic, artificial intelligence,
machine learning and data mining to diagnostics and anomaly detection, software diagnostics
engineering and diagnostics-driven development, diagnostics workflow and interaction. Recent
interest areas also include cloud native computing, security, automation, functional programming, and
applications of category theory to software development and big data.
6
Presentation Slides and Transcript
7
8
Hello, everyone, my name is Dmitry Vostokov, and I teach this training course. The third edition of this
course covers the M2 ARM64 platform and LLDB debugger. GDB debugger exercises were removed. If
you are interested in GDB and ARM64 there’s a Linux course available.
9
The prerequisites are hard to define. Some of you have software development experience and some
do not. However, one thing is certain: to get most of this training you are expected to have basic
troubleshooting experience. Another thing I expect you to be familiar with is hexadecimal notation
and that you have seen or can read programming source code in some language, preferably in C. The
ability to read assembly language has some advantages but for most parts not necessary for this
training. I hope to provide all the necessary explanations in this edition. Windows or, better, Linux
memory dump analysis experience may really help here and ease the transition but is not absolutely
necessary. If you have attended training or read books Accelerated Windows Memory Dump Analysis
or Accelerated Linux Core Dump Analysis, you would find a similar approach here.
10
Our primary goal is to learn core dump analysis in an accelerated fashion. So first we review
absolutely essential fundamentals necessary for core dump analysis. Also, this training is about user
process core dump analysis and not about kernel core dump analysis. An additional goal is to leverage
Windows or Linux debugging and memory dump analysis experience you may have.
11
For me, there were many training formats to consider, and I decided that the best way is to
concentrate on hands-on exercises. Specifically, for this training, I developed 12 of them, and they
utilize the same pattern-oriented approach I used in Accelerated Windows Memory Dump Analysis
and Accelerated Linux Core Dump Analysis training courses.
12
This is a roughly planned schedule. This edition also includes ARM64 disassembly and I discuss it
after we do some core dump analysis to have some context and possible “aha” moments.
13
Now I show you some pictures. We use 64-bit examples because all Mx Macs are 64-bit.
14
If you are coming from Windows or Linux background, you find fundamentals almost the same. For
every process, the macOS memory range is divided into kernel space part, user space part, and 4GB
non-accessible part to catch null pointers. This non-accessible region is different from Windows
where it is only a few Kb or Linux where it is 64KB on my Debian system. I follow the long tradition to
use red color for the kernel and blue color for the user part. Please note that there is a difference
between space and mode. Mode is the execution privilege attribute, for example, code running in
kernel space has higher execution privilege than code running in user space. However, kernel code
can access user space and access data there. We say that such code is running in kernel mode. On the
contrary, the application code from user space is running in user mode and because of its lower
privilege, it cannot access kernel space. This prevents accidental kernel modifications. Otherwise, you
could easily crash your system. I put addresses on the right. This uniform memory space is called
process virtual space because it is an abstraction that allows us to analyze core dumps without
thinking about how it is all organized in physical memory. When we look at process dumps, we are
concerned with virtual space only. In this training, we would only see user space. The picture is
conceptually unchanged from Intel Macs and it is consistent with what we see in spindump and
vmmap output.
15
When an app is loaded all its referenced frameworks, and dynamic libraries are mapped to virtual
memory space. Different sections of the same file (like code and data) may be mapped into a different
portion of memory. In contrast, modules in Windows and shared libraries in Linux are organized
sequentially in virtual memory space. A process then is setup for running, and a process ID is assigned
to it. If you run another such app, it will have a different virtual memory space.
16
When we save a process core memory dump, a user space portion of the process space is saved
without any kernel space stuff. However, we never see such large core dumps unless we have memory
leaks. This is because process space has gaps unfilled with code and data. These unallocated parts are
not saved in a core dump. However, if some parts were paged out and reside in a page file, they are
usually brought back before saving a core dump. On the Mx platform, dumps are about 3GB.
17
Now we come to another important fundamental concept in macOS core dump analysis: thread. It is
basically a unit of execution, and there can be many threads in a given process. Every thread just
executes some code and performs various tasks. Every thread has its ID. In this training, we also learn
how to navigate between process threads. Note that threads transition to kernel space via the
libsystem_kernel dynamic library similar to ntdll in Windows or the libc in Linux. Threads additional to
the main thread (POSIX Threads) originate from the libsystem_c dynamic library.
18
Every thread needs a temporary memory region to store its execution history and temporary data.
This region is called a thread stack. Please note that the stack region is just any other memory region,
and you can use any LLDB data dumping commands there. We will also learn how to get thread stack
region address range. Examining raw stack data can give some hints about the past app behavior: the
so-called Execution Residue pattern.
19
Now we explain thread stack traces. Suppose we have source code where FunctionA calls FunctionB at
some point and FunctionB calls FunctionC and so on. This is a thread of execution. If FunctionA calls
FunctionB, you expect the execution thread to return to the same place where it left and resume from
there. This is achieved by saving a return address in the thread stack region. So every return address
is saved and then restored during the course of a thread execution. Although the memory addresses
grow from top to bottom in this picture, return addresses are saved from bottom to top. This might
seem counter-intuitive to all previous pictures, but this is how you would see the output from LLDB
commands. What LLDB does when you instruct it to dump a backtrace from a given thread is to
analyze the thread raw stack data and figure out return addresses, map them to a symbolic form
according to symbol files and show them from top to bottom. Note that FunctionD is not present in the
raw stack data on the left because it is a currently executing function called from FunctionC. However,
FunctionC called FunctionD, and the return address of FunctionC was saved. In the box on the right, we
see the result of the LLDB command.
20
The difference from WinDbg here is that the return address is on the same line for the function to
return (except for FunctionD, where the address is the next instruction to execute) whereas in
WinDbg it is for the function on the next line. There’s not difference from GDB except offsets.
21
Here I’d like to show you why symbol files are important and what stack traces you get without them.
Symbol files just provide mappings between memory address ranges and associated symbols like the
table of contents in a book. So in the absence of symbols, we are left with bare addresses that are
saved in a dump. For example, without AppA symbols, we have the output shown in the box on the
right.
22
Now we talk about access violation exceptions. During the thread execution, it accesses various
memory addresses doing reads and writes. Sometimes memory is not present due to gaps in virtual
address space or different protection levels like read-only or no-execute memory regions. If a thread
tries to violate that, we get an exception that is also translated to a traditional UNIX signal. Certain
regions are forbidden to read and write such as the first 4GB. If we have such an access violation
there, then it is called a NULL pointer access. Note that every thread can have an exception (a victim
thread) and it often happens that there are multiple exceptions. It is also sometimes the case that code
can catch these exceptions preventing a user from seeing error messages. Such exceptions can
contribute to corruption, and we call then hidden.
23
However, not all exceptions happen from invalid access. Many exceptions are generated by the code
itself when it checks for some condition, and it is not satisfied, for example, when the code checks a
buffer or an array to verify whether it is full before trying to add more data. If it finds it is already full,
the code throws an exception translated to SIGABRT. We would see that in one of our practice
examples when C++ code throws a C++ exception. Such exceptions are usually called runtime
exceptions.
24
A few words about logs, checklists, and patterns. Core memory dump analysis is usually an analysis of
a text for the presence of diagnostic patterns. We run commands, they output text, and then we look at
that textual output, and when we find suspicious diagnostic indicators, we execute more commands.
Here pattern and command checklists can be very useful.
25
26
Core Dump Collection
27
28
Here I’d like to show you how to collect core dumps because by default this option is switched off on
macOS and there are further complications compared to previous versions.
29
You should also have a directory /cores where all process core dumps are stored.
30
31
32
ARM64 Disassembly
33
34
35
There are 31 general registers from X0 and X30, with some delegated to specific tasks such as
addressing stack frames (Frame Pointer, FP, X29) and return addresses, the so-called Link Register
(LR, X30). When you call a function the return address of a caller is saved in LR, not on the stack as in
Intel/AMD x64. The return instruction in a callee will use the address in LR to assign it to PC and
resume execution. But if a callee calls other functions the current LR needs to be manually saved
somewhere, usually on the stack. There’s Stack Pointer, SP, of course. To get zero values there’s the
so-called Zero Register, XZR. All X registers are 64-bit, and 32-bit lower parts are addressed via the W
prefix. Additionally, when disassembly runtime functions you may see the usage of 128-bit SIMD
registers(and also 64-bit floating point registers). These may be used to speed up some common
operations such as zero-initializing structures. The References slide provides links to the ARM64
instruction set architecture. Next, we just briefly look at some aspects related to our exercises.
36
This slide shows a few examples of CPU instructions that involve operations with registers, for
example, moving a value and doing arithmetic. The direction of operands is the same as in Intel x64
disassembly flavor if you are accustomed to WinDbg and GDB on Windows and Linux platforms. It is
equivalent to an assignment. BLR is a call to some function whose address is in the register. BL means
Branch and Link.
37
Before we look at operations with memory, let's look at a graphical representation of memory
addressing. A thread stack is just any other memory region, so instead of SP and X29 (FP), any other
register can be used. Please note that the stack grows towards lower addresses so to access the
previously pushed values you need to use positive offsets from SP.
38
Constants are encoded in instructions but if we need arbitrary values, we must get them from
memory. Square brackets are used to show memory access relative to an address stored in some
register. There’s also an option to adjust the value of the register after load, the so-called
Postincrement, which can be negative. As we see later, loading pairs of registers can be useful.
39
Storing operand order goes in the other direction compared to other instructions. There’s a possibility
to Preincrement the destination register before storing values.
40
Because the size of every instruction is 4 bytes (32 bits) it is only possible to encode a part of a large
4GB address range, either as a relative offset to the current PC or via ADRP instruction. Goto (an
unconditional branch) is implemented via the B instruction. Function calls are implemented via the
BL (Branch and Link) instruction. For conditional branches please look at the official documentation
provided on the References slide. We don’t use these instructions in our exercises.
41
When a function is called from the caller, a callee needs to do certain operations to make a room for
local variables on the thread stack and save LR if there are further calls in the function body. There
are different ways to do that and the assembly language code on the left is one of them. I use a
different color in the diagram on the right to highlight the updated SP and X29 (FP) values before
proc2 is called. Please also note an example of zero register usage. For simplicity of illustration, I only
use 64-bit values.
42
You may have noticed on the previous diagram that the new X29 (FP) points to the X29 of the caller
and below the previous X29 is the return address of the caller. So, if you know either the return
address in LR or X29 you can reconstruct the stack trace if the compiler follows the preceding
function prolog convention.
43
Another feature we encounter in our exercises is pointer authentication. The References slide
contains the link to the PAC paper, so I only mention that some parts of a pointer value (higher bits)
contain verification codes. Memory dumping commands automatically ignore these parts to show the
real address value in order to provide a symbolic reference if any. However, these bits may be a part
of the real virtual address (or considered such erroneously) as shown on this slide. In this case, you
can use the so-called typed memory print in LLDB which doesn’t remove these high bits.
44
Practice Exercises
45
Other documents randomly have
different content
“yellows” can be considered as certain. The only sure way is to dig
out and burn every tree as soon as it is seen to be affected. This
plan has been followed in Michigan, where, between 1870 and
1880, the disease was very bad. Now hardly a case of “yellows”
can be found in many of the peach regions. Constant attention
and prompt action have proved successful, in this case, at least.
Blowing
Peddlers and some grocers will tell you, “this is as good as” or
“the same as Pearline.” IT’S FALSE—but what a puff for Pearline.
JAMES PYLE, New York.
“THE GEM”
This is a novelty of great
merit. The only objection to
the old variety is that it
sometimes grows tall and
scraggly, but the “Little
Gem” is of strong and dwarf
habit. The foliage which is of
a lustrous dark green is in
great abundance. The flowers
are produced in the greatest
profusion, being literally an
ever-bloomer, it will bloom
freely all summer in the open ground, in September it can be
lifted and potted and will continue blooming all winter. The “Little
Gem” Calla will continue to grow and bloom for years without
ceasing, and the quantity of flowers which a large plant will
produce is astonishing, the flowers are snowy white in color, and
of good size, it seldom grows higher than fifteen inches. Price for
plants that will bloom this season, 30 cents each. For $1.00 I
will mail 5 plants to one address.
ONLY $1.00 will buy any one of the collections named below,
delivered safely by mail, postpaid, to any address. The collections
are all fine, strong plants of the best varieties and are marvels of
cheapness. Every plant is plainly labeled, and there are no two
varieties alike in the same collection.
FOR $1.00 I will mail FREE 20 Prize-winning
Chrysanthemums; 20 fine Single and Double Flowering
Geraniums for $1.00; 20 Choice Ever-Blooming Carnations
for $1.00; 20 Flowering Begonias for $1.00; 20 Assorted
Flowering Plants for $1.00; 20 Fancy Leaved Coleus for
$1.00.
For $5.00 you can select any six of the above
Collections.
To every one who sends an order from this advertisement and
mentions this magazine, we will send FREE a valuable plant.
ORDER NOW and ask for our CATALOGUE of BEAUTIFUL
FLOWERS for 1894.
CHARLES A. REESER, INNISFALLEN GREENHOUSES,
SPRINGFIELD, O.
D. and C.
ROSES
Will grow anywhere, with a little sunshine, water, and care.
You can learn how to grow them, and every other flower of
import, from our
New Guide to Rose Culture
for 1894. If you so request, we will send free, this book and a
copy of our Floral Magazine, ‘Success with Flowers.’
The Dingee & Conard Company,
West Grove, Pa.
ELEGANT PALMS
From India and the Isles of the Sea. 5 Glorious Plants, different
sorts, post paid, 60c. These will grow and flourish everywhere.
PALM SEED.
It is child’s-play to make them grow. Send 5c. postage for our
great catalogue, (130 pages); or catalogue and one large package
of 5 different kinds of Palm seed, free for 20c. postage. 100
packages $10. A child can sell 100 packages in two evenings after
school and make $5.00.
Banquet Strawberry.
Equal to wild berry in flavor. CROSBY PEACH, frost proof.
Fruits every year. Colored Plates. Full descriptions. Free
Catalogue. All fruits. Write at once. HALE BROS., South
Glastonbury, Conn.
1.D. The copyright laws of the place where you are located also
govern what you can do with this work. Copyright laws in most
countries are in a constant state of change. If you are outside
the United States, check the laws of your country in addition to
the terms of this agreement before downloading, copying,
displaying, performing, distributing or creating derivative works
based on this work or any other Project Gutenberg™ work. The
Foundation makes no representations concerning the copyright
status of any work in any country other than the United States.
1.E.6. You may convert to and distribute this work in any binary,
compressed, marked up, nonproprietary or proprietary form,
including any word processing or hypertext form. However, if
you provide access to or distribute copies of a Project
Gutenberg™ work in a format other than “Plain Vanilla ASCII” or
other format used in the official version posted on the official
Project Gutenberg™ website (www.gutenberg.org), you must,
at no additional cost, fee or expense to the user, provide a copy,
a means of exporting a copy, or a means of obtaining a copy
upon request, of the work in its original “Plain Vanilla ASCII” or
other form. Any alternate format must include the full Project
Gutenberg™ License as specified in paragraph 1.E.1.
• You comply with all other terms of this agreement for free
distribution of Project Gutenberg™ works.
1.F.
Most people start at our website which has the main PG search
facility: www.gutenberg.org.
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
ebooknice.com