Malware Analysis

Professional

Anti-Reversing Tricks: Part 3

S e c t i o n 0 2 | M o d u l e 1 2
© Caendra Inc. 2020
All Rights Reserved
Table of Contents

MODULE 12 | ANTI-REVERSING TRICKS: PART 3

12.1 Introduction
12.2 Software vs. Hardware Breakpoints
12.3 Software Breakpoint Detection
12.4 Hardware Breakpoint Detection
12.5 Ring0 Debuggers & System Monitoring Tools Detection
12.6 Structured Exception Handling (SEH)
12.7 Unhandled Exception Filter
12.8 VM Detection
12.9 Conclusion

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.2

Getting Started

Tools:
• Olly Debugger v1.10

Target:
• RE_Lab_12.zip

www.ollydbg.de MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.3

 12.1

Introduction

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.4

12.1 Introduction

In this module we will continue discussing a few more

common Anti-RE tricks.

We will also start discussing how an application can detect

software and hardware breakpoints, and we will talk about
the functional differences between the two.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.5

12.1 Introduction

Furthermore, will be demonstrating an easy way to detect

Ring0 debuggers or system monitoring tools, and will also
demonstrate how exceptions can be used as an anti-
reversing trick.

Finally, we will demonstrate some ways to detect some

well-known virtualization environments..

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.6

12.1 Introduction

Layout:
i) Anti-RE trick Category
a) Category-related trick example
b) Another related example
c) etc.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.7

 12.2

Software vs. Hardware

Breakpoints

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.8

12.2 Software vs. Hardware Breakpoints

Before going into demonstrating how it is possible to detect

these two types of breakpoints, let’s explain the differences
between them.

A software breakpoint is placed by substituting the byte

originally located at that memory address by a software
interrupt - specifically the INT 3h interrupt of which the
opcode is the 0xCC.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.9
12.2 Software vs. Hardware Breakpoints

When the execution ‘hits’ an INT 3h instruction (also called

a trap to the debugger) a software breakpoint exception is
raised (exception code: 80000003h) and if the process is
being debugged, the debugger will force the execution to
stop there.

So, as you can see software breakpoints are quite intrusive

since they modify the code in memory.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.10
12.2 Software vs. Hardware Breakpoints

Furthermore, software breakpoint capabilities are also

limited, since they can only be used for code under
execution and not for memory access monitoring.

The only advantage of software breakpoints is that you can

place as many as you want during debugging.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.11

12.2 Software vs. Hardware Breakpoints

On the other hand, hardware breakpoints use the debug

registers DR0-DR3. They can be used for different types of
memory access monitoring and not just to break the
execution on a specific instruction. It is possible for
example, to set a hardware breakpoint on access on a
specific memory area inside the virtual address space of
the process under analysis, which will be triggered when
that memory address is accessed for read/write.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.12

12.2 Software vs. Hardware Breakpoints

As nothing is perfect, hardware breakpoints have their own

limitations, since it is not possible to have more than four
(per thread) at the same time.

This makes sense since, as you already know for each

thread there is a CONTEXT structure describing the state of
the CPU of that thread, or in other words the values of the
CPU Registers necessary in order to keep track of the
execution state for that specific thread.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.13
12.2 Software vs. Hardware Breakpoints

So, when we set a hardware breakpoint, we update the

status of a Debug register (DR0-DR3) for that thread under
execution with the corresponding virtual address where the
hardware breakpoint will apply. This means that hardware
breakpoints are also per-thread specific. Unless, of course
the debugger we are using is smart enough to update the
corresponding Debug register value for all the active thread
of the debugging process.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.14

12.2 Software vs. Hardware Breakpoints

As in the case of software breakpoints, a special exception

is raised to signal the access of a specific memory location
for which a hardware breakpoint has been set (on
execution, on read/write, etc.).

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.15

12.2 Software vs. Hardware Breakpoints

One big advantage of using hardware breakpoints over

software breakpoints is the fact that they don’t do any code
modifications in memory.

This makes them very suitable to use in cases where self-

modifying code is present (or other tricks that detect
software breakpoints by calculating checksums of code
blocks.)
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.16
 12.3

Software Breakpoint
Detection

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.17

12.3 Software Breakpoint Detection

…a typical example

push ‘kernel32.dll’
call LoadLibrary ;get imagebase of kernel32.dll
push ‘VirtualProtect’
push eax
GetProcAddress ;get address of ‘VirtualProtect’ API inside
exported by kernel32.dll
cmp byte ptr ds:[eax], 0xCC ;check if there is a breakpoint set
there
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.18
 12.4

Hardware Breakpoint
Detection

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.19

12.4 Hardware Breakpoint Detection

The most common way involves the usage of 2 Windows

APIs:
1. OpenThread → Get a handle to the desired thread.

2. GetThreadContext → Read the current thread context

and locate the values of the Debug Registers used to
store HW BPs, DR0-DR3.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.20

12.4 Hardware Breakpoint Detection

A more obscure way involves the use of exceptions:

Generate an exception → then, once inside the exception
handler, find the context record location from ESP + 0x0C
→ and check the values of DR0-DR3.

NOTE: DR0-DR3 will be zero if no HW BPs are set.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.21

 12.5

Ring0 Debuggers &

System Monitoring
Tools Detection

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.22

12.5 Ring0 Debuggers & System Monitoring Tools
Detection

The most common use involves the usage of the CreateFile

Windows API.

Such types of tools use drivers with which to communicate

using their own named devices:
\\.\NTICE → Softice (Windows NT)
\\.\FILEM → FileMon (Windows NT)
\\.REGSYS → RegMon (Windows NT)
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.23
12.5 Ring0 Debuggers & System Monitoring Tools
Detection

So, if we manage to obtain a valid handle to a specific

device like these above, then we actually reveal the
presence of the driver belonging to that tool.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.24

 12.6

Structured Exception
Handling (SEH)

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.25

12.6 Structured Exception Handling (SEH)

Finally, we can discuss debugger detection through

exception generation.

This is mainly used for debugger detection but can be used

for redirecting the execution flow under certain
circumstances; it is a nice logic obfuscation tool.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.26

12.6 Structured Exception Handling (SEH)

Using INT 3h example:

push _exception_handler
push dword fs:[0]
mov dword fs:[0],esp
INT 3h ;if a debugger is present, it will handle the exception believing it is a SW BP set by the
user.
mov eax, 1
jmp _continue

_exception_handler:
xor eax,eax ; if we arrive here debugger not present.

_continue:
;more code here

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.27

12.6 Structured Exception Handling (SEH)

So, what basically happens in this example is that we set

our own exception handler and then we attempt to execute
an INT 3h instruction which is the equivalent of a software
breakpoint.

Depending on the debugger’s settings, generally a debugger

will think that this is a software breakpoint set by the user
so it will handle the generated exception itself and set the
EIP to point to the next instruction (‘mov eax,1’).
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.28
12.6 Structured Exception Handling (SEH)

The point is, if the process is not being debugged, since an

exception is raised, the execution should reach the
exception handler from where we can set the EIP
accordingly.

You might think this piece of code as being part of a

Boolean function which returns 1 if a debugger has been
detected or 0 if not.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.29
12.6 Structured Exception Handling (SEH)

TIP: Instead of INT3, a call to DebugBreak Windows API

can be used which executes an INT3 instruction.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.30

 12.7

Unhandled
Exception Filter

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.31

12.7 Instruction Permutations

This is a powerful anti-reversing technique that uses a

specific exception handler (normally used when there are
no appropriate handlers to handle an exception.)

In this case, if a process is being debugged, after the

execution of the UnhandledExceptionFilter API, the process
will exit instead of continuing execution which, in the
context of anti-reversing purposes, is very suitable.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.32
12.7 Instruction Permutations

When the UnhandledExceptionFilter API is called it will

itself make a call (through a subroutine) to the
ZwQueryInformationProcess API asking for
ProcessDebugPort information which will return
0xFFFFFFFF if the process is being debugged by a Ring3
debugger, and 0x0 if it is not.

Basically, according to MSDN documentation, if it returns a

non-zero value, the process is being debugged by a Ring3
debugger.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.33
12.7 Instruction Permutations

We can force that subroutine to always return 0x00, in order

for the SetUnhandledExceptionFilter API to be called and
transfer execution to the custom exception handler.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.34

12.7 Instruction Permutations

In other words, the trick relies on the fact that we can set a
custom exception handler through the
SetUnhandledExceptionFilter API. When an exception is
raised, the UnhandledExceptionFilter will be called and will
also check if the process is being debugged.

According to MSDN, this API is designed to pass the

exception to the debugger if the process is being
debugged. Once this is done, the process will be
terminated.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.35
12.7 Instruction Permutations

However, if we trick the application into thinking that the

process is not being debugged, then the execution will be
transferred to our custom exception handler and we will be
able to continue debugging the application.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.36

 12.8

VM Detection

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.37

12.8 VM Detection

The use of virtualization environments for application

testing and malware analysis it is a very common practice
nowadays.

Some application authors, either for a malicious application

or not, don’t want to allow the execution of their application
inside a VM such as the well-known VMware or VirtualBox
in order to avoid time trial bypasses (in the case of
shareware), etc.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.38
12.8 VM Detection

Some commercial anti-piracy protectors for applications

also include the capability to terminate the application it is
running in a VM.

There are many ways to achieve this, but some of them are
well-known and are widely used.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.39

12.8 VM Detection

VMware detection
mov eax, ‘VMXh’  magic number
mov ebx, 0
mov ecx, 0Ah  set function number / 0Ah = get VMware
version
mov edx, 5658h  port number number used to communicate
with VMware
in eax, dx  read a dword from that port
cmp eax, ebx  if VMware is present EAX == EBX
je __VMware_detected
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.40
12.8 VM Detection

The piece of code on the following slide uses a logical port

that is used for communication between the virtualized
environment and VMware itself in order to get information
about the VMware version used.

The concept behind this trick is that the ‘in eax,dx’

instruction is a privileged one. We can’t usually execute this
instruction from usermode (Ring3).
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.41
12.8 VM Detection

However, the virtual CPU of VMware will allow the

execution of this instruction to communicate with VMware
itself.

This trick is always used along with an exception handler

since if the application is not running inside VMware, a
privileged instruction exception will be raised (exception
code: C0000096h) and will cause the application to crash
without the handler in place.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.42
12.8 VM Detection

VirtualPC detection
mov ebx, 0
mov eax, 1
db 0Fh, 3Fh, 7, 0Bh // VPC Call
cmp ebx, 0
je __VPC_detected

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.43

12.8 VM Detection

The virtual CPU in VirtualPC can decode this set of bytes

‘0Fh, 3Fh, 7, 0Bh’ as special call instructions which of
course don’t resolve to valid instructions from a native x86
CPU.

Attempting to execute this virtual instruction in a physical

system will cause an illegal instruction exception
(exception code: C000001Dh). For this reason this trick has
to be used along with an associated exception handler.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.44
12.8 VM Detection

VirtualBox detection
A very simple and easy way to detect VirtualBox is through the
window class name of a tray icon that it places in the taskbar.

push 0
push ‘VBoxTrayToolWndclass’
call FindWindowA
test eax,eax
jnz _VboxDetected ; if we managed to obtain a valid window
handle VBox was detected
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.45
12.8 VM Detection

Of course there are many other ways for detecting a VM

environment so you shouldn’t only limit yourself to these.

Doing some research is always a good thing.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.46

 12.9

Conclusion

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.47

12.9 Conclusion

Throughout these three modules on Anti-Reversing tricks,

we’ve managed to demonstrate some common ways used
by code authors to detect a reversing tool or a VM
environment.

However, there are many other tricks in the wild that you
might have to deal with and many variations of those we
talked about.
MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.48
12.9 Conclusion

The main goal of these modules was to make you aware of

some possible obstacles you will most probably come
across in the future, especially as a malware analyst.

In the next module, we will be discussing the most

important topic regarding code anti-analysis techniques:
‘obfuscation’.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.49

VIDEO
Check out the video on Anti-
Reversing Tricks: Part 3!

To ACCESS your video, go

to the course in your
members area and click the
resources drop-down in the
appropriate module line.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.50

LAB

Put what you’ve learned to

practice with the
RE_Lab_12.zip!

To ACCESS your lab, go to

the course in your
members area and click
the resources drop-down
in the appropriate module
line, your file will then
download.

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.51

References

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.52

 References
Here’s a list of all references linked or used in this course.
Olly Debugger v1.10
http://www.ollydbg.de/

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.53

 Videos & Labs
Here’s a list of all videos and labs in this module. To ACCESS, go to the
course in your members area and click the resources drop-down in the
appropriate module line.

Anti-Reversing Tricks: Part 3

RE_Lab_12.zip

MAPv1: Section 02, Module 12 - Caendra Inc. © 2020 | p.54