Full download Fundamentals of Trace and Log Analysis A Pattern-Oriented Approach to Monitoring, Diagnostics, and Debugging 1st Edition Dmitry Vostokov pdf docx
Full download Fundamentals of Trace and Log Analysis A Pattern-Oriented Approach to Monitoring, Diagnostics, and Debugging 1st Edition Dmitry Vostokov pdf docx
https://ebookmass.com
https://ebookmass.com/product/foundations-of-arm64-linux-debugging-
disassembling-and-reversing-dmitry-vostokov/
testbankdeal.com
Foundations of ARM64 Linux Debugging, Disassembling, and
Reversing: Analyze Code, Understand Stack Memory Usage,
and Reconstruct Original C/C++ Code with ARM64 1st Edition
Dmitry Vostokov
https://ebookmass.com/product/foundations-of-arm64-linux-debugging-
disassembling-and-reversing-analyze-code-understand-stack-memory-
usage-and-reconstruct-original-c-c-code-with-arm64-1st-edition-dmitry-
vostokov/
testbankdeal.com
https://ebookmass.com/product/principles-of-bloodstain-pattern-
analysis-theory-and-practice-practical/
testbankdeal.com
https://ebookmass.com/product/using-r-for-data-analysis-in-social-
sciences-a-research-project-oriented-approach-li/
testbankdeal.com
https://ebookmass.com/product/the-strengths-model-a-recovery-oriented-
approach-to-mental-health-services/
testbankdeal.com
Fundamentals
of Trace and
Log Analysis
A Pattern-Oriented Approach to
Monitoring, Diagnostics, and
Debugging
—
Dmitry Vostokov
Fundamentals of
Trace and Log
Analysis
A Pattern-Oriented Approach
to Monitoring, Diagnostics,
and Debugging
Dmitry Vostokov
Fundamentals of Trace and Log Analysis: A Pattern-Oriented Approach to
Monitoring, Diagnostics, and Debugging
Dmitry Vostokov
Dalkey, Dublin, Ireland
Introduction����������������������������������������������������������������������������������������xv
Chapter 1: Introduction������������������������������������������������������������������������1
Software Trace/Log�����������������������������������������������������������������������������������������������3
Process�����������������������������������������������������������������������������������������������������������������4
Thread�������������������������������������������������������������������������������������������������������������������4
Adjoint Thread�������������������������������������������������������������������������������������������������������5
Component/Module/Source����������������������������������������������������������������������������������7
File and Function��������������������������������������������������������������������������������������������������8
Trace Message������������������������������������������������������������������������������������������������������9
Stack Trace�����������������������������������������������������������������������������������������������������������9
Minimal Trace Graphs�����������������������������������������������������������������������������������������10
Pattern-Oriented Analysis�����������������������������������������������������������������������������������12
Pattern Classification������������������������������������������������������������������������������������������13
Summary������������������������������������������������������������������������������������������������������������14
v
Table of Contents
vi
Table of Contents
Focus of Tracing��������������������������������������������������������������������������������������������������55
Event Sequence Order����������������������������������������������������������������������������������������57
Trace Frames������������������������������������������������������������������������������������������������������58
Summary������������������������������������������������������������������������������������������������������������60
vii
Table of Contents
Linked Messages������������������������������������������������������������������������������������������������90
Gossip�����������������������������������������������������������������������������������������������������������������93
Abnormal Value���������������������������������������������������������������������������������������������������94
Message Context������������������������������������������������������������������������������������������������95
Marked Messages�����������������������������������������������������������������������������������������������97
Fiber Bundle��������������������������������������������������������������������������������������������������������98
Incomplete History��������������������������������������������������������������������������������������������100
Opposition Messages����������������������������������������������������������������������������������������100
Summary����������������������������������������������������������������������������������������������������������101
viii
Table of Contents
Index�������������������������������������������������������������������������������������������������139
ix
Visit https://ebookmass.com
now to explore a rich
collection of eBooks and enjoy
exciting offers!
About the Author
Dmitry Vostokov is an internationally recognized expert, speaker,
educator, scientist, inventor, and author. He founded the pattern-
oriented software diagnostics, forensics, and prognostics discipline
(Systematic Software Diagnostics) and Software Diagnostics Institute
(DA+TA: DumpAnalysis.org + TraceAnalysis.org). Vostokov has also
authored multiple books on software diagnostics, anomaly detection
and analysis, software and memory forensics, root cause analysis and
problem solving, memory dump analysis, debugging, software trace and
log analysis, reverse engineering, and malware analysis. He has over 30
years of experience in software architecture, design, development, and
maintenance in various industries, including leadership, technical, and
people management roles. In his spare time, he presents multiple topics
on Debugging.TV and explores Software Narratology and its further
development as Narratology of Things and Diagnostics of Things (DoT),
Software Pathology, and Quantum Software Diagnostics. His current
interest areas are theoretical software diagnostics and its mathematical
and computer science foundations, application of formal logic, artificial
intelligence, machine learning, and data mining to diagnostics and
anomaly detection, software diagnostics engineering and diagnostics-
driven development, diagnostics workflow, and interaction. Recent
interest areas also include cloud-native computing, security, automation,
functional programming, applications of category theory to software
development and big data, and artificial intelligence diagnostics.
xi
About the Technical Reviewer
Vijender Singh, a multicloud professional
with over six years of expertise, holds an MSc
with distinction from Liverpool John Moores
University, where his research centered on
keyphrase extraction. He boasts an impressive
collection of certifications, including MLPE
GCP, five Azure certifications, two AWS
certifications, and TensorFlow certification.
Vijender’s role as a technical reviewer for
numerous books reflects his commitment to
improving the future.
LinkedIn: www.linkedin.com/in/vijendersingh412
xiii
Introduction
This book will help you analyze traces and logs from different software
environments and communicate analysis results using a pattern language
that covers everything from a small debugging log to a distributed trace
with billions of messages from hundreds of computers, thousands of
software components, threads, and processes.
The book begins with the basic terminology of operating systems and
programming, the foundation for understanding trace and log analysis.
It then talks about patterns that help describe problems from a user’s
view and patterns for errors and failures. Then, the book covers a range of
trace patterns that group messages and explores how logs depict software
activities. It even looks at specific message patterns and how they connect
in a single trace. Toward the end, the book goes over patterns for multiple
traces and logs and how to understand them as data. In this way, you can
use similar methods to find problems across a wide variety of software. It
also guides you on analyzing issues on systems such as Windows, macOS,
Linux, Android, iOS, and other types of computers, including those in
networks and the Internet of Things, regardless of their system differences.
Upon completing this book, you will be able to navigate the
complexities of trace and log analysis and apply uniform diagnostics and
anomaly detection pattern language across diverse software environments
to help you troubleshoot, debug, and fix issues.
The book will be useful for software technical support engineers,
system and network administrators, software developers, testers, DevOps
and DevSecOps, digital forensics and malware analysts, security incident
response engineers, data analysts, and data mining practitioners.
xv
CHAPTER 1
Introduction
We start this chapter with a few words about the need for this book. Almost
20 years ago, I started doing Windows software diagnostics full time as
a member of the technical support and escalation team at a large global
software vendor. In addition to crash and hang dump analysis, the job
required analysis of software traces similar to that of Process Monitor1
log format with messages from hundreds of processes and thousands of
threads totaling millions of lines (Figure 1-1). Gradually I became aware
that we need a similar pattern-driven system as I devised for memory
dump analysis.2 However, after a few patterns, such as Periodic Error, I
was stuck devising more. At this time, through my voracious independent
reading, I accidentally became acquainted with narratology,3 a discipline
that studies narration and narrative stories. So this became the foundation
for what I later named Software Narratology, a new approach to the study
of software narrative, stories of computation. Viewing software traces as
narratives helped devise general analysis patterns to structure trace and
log analysis independent of OS and products. Although this book teaches
some analysis patterns in a Windows context, you can apply them to your
specific environment and product domain problems. Some examples are
1
https://learn.microsoft.com/en-gb/sysinternals/downloads/procmon
2
Dmitry Vostokov, Encyclopedia of Crash Dump Analysis Patterns: Detecting
Abnormal Software Structure and Behavior in Computer Memory, Third Edition,
2020 (ISBN-13: 978-1912636303)
3
https://en.wikipedia.org/wiki/Narratology
illustrated with Process Monitor and Event Tracing for Windows4 because
they are widely used and not tied to specific products. In addition to native
logging, Linux has LTTng5 tracing. All these tracing frameworks produce
trace and log files with conceptually similar formats.
4
https://learn.microsoft.com/en-us/windows/win32/etw/
about-event-tracing
5
https://lttng.org/
6
Dmitry Vostokov, Trace, Log, Text, Narrative, Data: An Analysis Pattern Reference
for Information Mining, Diagnostics, Anomaly Detection, Fifth Edition, 2023
(ISBN-13: 978-1912636587)
2
Chapter 1 Introduction
Software Trace/Log
What is a software trace or log, actually? For our purposes, it is just a
sequence of formatted messages sent from running software. They are
usually arranged by time and can be considered as a software narrative
story (Figure 1-2). In this training, we confine ourselves to the analysis of
such logs and what patterns to look for.
3
Chapter 1 Introduction
Process
An operating system process is a container for resources such as memory,
files, and synchronization objects. Even the OS kernel itself can be
considered a process itself (in Windows, it is usually called just System).
Each process has its own process identifier, PID (process ID). In Windows,
it belongs to a user session; for example, there can be several users logged
into Windows. Each process has its own image name, such as notepad.
exe, and a list of associated loaded DLL (dynamic-link library) modules
(or shared libraries in Linux/macOS). An image name is also a module.
It is important to remember that there can be several processes running,
each having the same image name, for example, two instances of notepad.
exe. The list of DLLs in both instances, most of the time, is identical. At the
same time, it is possible that one image name covers completely different
processes because, on the startup, a process loads different modules
for different purposes. Here an example is the svchost executable. On a
running Windows system, you can find many such svchost processes.
When we analyze software logs, we can filter messages related to a specific
PID or image name to find any abnormal behavior according to the
expected message flow. A typical example here: after the middle of the full
trace, we no longer see any more messages from the specific PID, not even
any termination or graceful process end messages.
Thread
In Windows and macOS, a thread is an execution unit with its own ID and
is owned by some process, for example, calculator.exe. In Linux, threads
are almost the same as processes but share the same virtual memory.
Remember that trace messages come from some thread because we need
to execute some code to emit a trace message. Each thread is executed on
some CPU and, in general, can have its CPU changed during execution
history. Filtering by threads, for example, allows us to find any anomalous
4
Chapter 1 Introduction
Adjoint Thread
If a thread is a linear ordered flow of activities associated with a particular
TID (thread ID), as seen from a trace message perspective through time,
we can also extend this flow concept and consider a linear flow of activities
associated with some other parameter such as PID, CPU, or message text.
Such messages have different TIDs associated with them but have some
chosen constant parameter or column value in a trace viewing tool. The
name adjoint comes from the fact that in threads of activity, TID stays the
same, but other message attributes vary; in adjoint threads, we have the
opposite. In Windows Process Monitor, we use exclusive and inclusive
filtering to form adjoint threads. By applying complex filtering criteria, we
get Adjoint Threads from other adjoint threads, for example, an adjoint
5
Chapter 1 Introduction
thread with specific PID and file activity formed after an inspection of
an adjoint thread with the same image name, such as POWERPNT.EXE
(Figure 1-4).
6
Visit https://ebookmass.com
now to explore a rich
collection of eBooks and enjoy
exciting offers!
Chapter 1 Introduction
Component/Module/Source
Trace messages come from a thread that belongs to a PID, but the code
to emit them resides in source code files. Some source code files, such as
static library code, can also be reused and included in different modules
(Figure 1-5). Such DLL modules or shared libraries can also be loaded
into different processes. Therefore, source or module (in a simpler) case is
another grouping of messages based on subsystem and functional division
that may include several source code files. By module or source filtering,
we can see subsystem activities.
7
Chapter 1 Introduction
// MainApp.c
foo () {
trace("foo: entry");
// do stuff
trace("foo: exit");
}
8
Random documents with unrelated
content Scribd suggests to you:
pronouncing that it is pleuritis, for example, or pneumonia, or the
like. But how often does it happen, that the complaint in question is
an aggregate of symptoms, produced by peculiarities of constitution,
and incidental circumstances, which, taken together, constitute an
ensemble which does not well admit of being referred to any one of
the general forms of disease described in our nosological systems?
Now, I say the most wonderful feature in the cases related by
Hippocrates, is that they are descriptive of the symptoms observed
in certain diseased individuals, instead of being, what most modern
cases are, symptoms drawn to correspond with certain ideal forms of
disease. What, in my opinion, likewise adds very much to the value
of these cases is, that (as Galen somewhere remarks in his
Commentary) the author never aimed to make his Books of
Epidemics a work on Therapeutics, and hence, in noting morbid
phenomena, his mind in not warped by any particular hypothesis,
nor by any selfish interest, in order to place some favorite mode of
practice, advocated by himself, in a favorable light. May I be
permitted here to remark, that the reader will be much struck with
our author’s admirable talent for describing the phenomena of
disease as they are actually presented to us, if he will compare the
case related by him in these two books with those of almost any
modern authority whatever;—for example, with those related by the
late Dr. James Hamilton, in his celebrated work on Purgative
Medicines. In the latter, you look in vain for the strongly-marked
features which present themselves in all the cases related by our
author,—for a description of the condition of the hypochondriac
region,—of the state of the animal heat in the extremities,—of the
minute characters of the alvine and urinary discharges,—of the
respiration,—of the patients’ position in bed,—and many other
symptoms, which are invariably noticed by Hippocrates. And what
reasonable person will venture to deny, that the symptoms I have
just now mentioned are most important features in every febrile
disease, and that no one can be said to have a sufficient view of
such a case, who does not take these into account? To confine our
attention at present to only one of these symptoms,—can it ever be
a matter of indifference what are the physical characters of so
important an excretion as the urine? that is to say, whether the
grosser particles of it, which usually fall to the bottom, be present in
the urine or not? Yet in all the seventeen cases related in the
modern work just now referred to, the characters of the urine are
not given in a single instance. And although the object of the writer
is to enforce his own peculiar views, as to the utility of purgative
medicines in this disease, he scarcely ever gives the minute
characters of the alvine discharges, as is uniformly the case with
Hippocrates; or if they are noticed at all, it is in so confused a
manner that the reader is at a loss to determine whether they are
produced by the disease, or by the medicines which have been
administered. For the issue of the case no obvious cause is stated,
but the reader is expected to draw the conclusion that, as purgatives
were freely given, and a considerable proportion of the cases did
well,—(agreeably to the hackneyed rule, post quod, ergo propter
quod,)—the purgatives brought about the fortunate result. Had the
cases been fully and circumstantially detailed, it might have been
found that, as in those related by Hippocrates, recovery was
preceded by a critical discharge of urine, accompanied with a
copious sediment; and then the more probable inference would have
been, that the amendment was referable to it, and not to the
purgative medicines which were administered. It is, I regret to say, a
notable example of the want of logical training in the education of
professional men, in the present age, that inferences regarding a
peculiar method of practice were allowed to be founded upon
narratives of observations so defective and one-sided as those I
refer to.