Sanjivani Rural Education Society’s

Sanjivani College of Engineering, Kopargaon-423 603

(An Autonomous Institute, Affiliated to Savitribai Phule Pune University, Pune)
NAAC ‘A’ Grade Accredited, ISO 9001:2015 Certified

Department of Computer Engineering

(NBA Accredited)

Subject- Digital Forensics (DF) [CO 315A)]

Unit 5 :- Software Reverse Engineering

Prof. Abhijit S. Bodhe

Assistant Professor
Department of Computer Engineering
E-mail :
[email protected]
Contact No: 7709 340 570
 Unit 5:- Software Reverse Engineering
• Software Reverse Engineering:
• Defend against software targets for viruses,
• worms and other malware,
• improving third party software library,
• identifying hostile codes-buffer overflow,
• provision of unexpected inputs.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 2

 Reverse engineering be used to improve the security

• Reverse engineering is the process of analyzing a system, software, or

device to understand how it works and what it does. It can be a
powerful tool for cyber security.
• Reverse engineering can be applied to understand the library's
functionality, identify security vulnerabilities, or undocumented
features that may be exploited.
• Reverse engineering covers a broad range of areas, including
decompiling and disassembling of executable files and libraries, and
analysis of system data.
• Once these vulnerabilities are identified, patches or security
enhancements can be applied.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 3

 Process of identifying a buffer overflow vulnerability
• Buffer overflow is a software coding error or vulnerability that can be
exploited by hackers to gain unauthorized access to corporate systems.
• For example, an attacker may introduce extra code, sending new instructions
to the application to gain access to IT systems.
• The process involves disassembling the application code to analyze how it
handles memory, particularly how it writes data to buffers.
• By examining the assembly instructions, one can identify if bounds checking
is absent or improperly implemented, leading to buffer overflows.
• Buffer overflow vulnerability happens when data written to a buffer exceeds
its size, which may overwrite important data or execute malicious code

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 4

 Static vs. Dynamic analysis techniques
• Static vs. Dynamic analysis techniques in identifying vulnerabilities in software:-
• Static analysis examines the code without executing it and can uncover
vulnerabilities like buffer overflows and memory leaks.
• Static analysis is a test of the internal structure of the application, rather
than functional testing.
• Dynamic analysis, on the other hand, involves running the software in
controlled conditions to observe its behavior, helping to identify runtime
issues and malicious payloads that only manifest during execution.
• Dynamic analysis adopts the opposite approach of static analysis and is
executed while a program is in operation.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 5

 Hidden malicious code
• These are fragments of code or malicious files hidden within legitimate websites
or third-party scripts or applications.
• step-by-step approach to reverse engineer an executable to find hidden malicious
code.
1.Begin by setting up a secure environment to contain potential malware.
2.Use a disassembler to convert the executable to assembly code.
3.Use static analysis tools (Eg. Fundl and RegCon) to scrutinize the code for suspicious patterns
and signatures.
4.Utilize dynamic analysis by running the program in a sandbox( keeping potentially malicious
program or unsafe code isolated from the rest of the organization's environment) to observe
behavior.
5.Use debugging tools to step through the code and inspect changes in system state and network activity.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 6

 Robustness of a software application
• Software robustness refers to the ability of a software system to continue
functioning correctly and reliably even in the face of unexpected or abnormal
inputs or situations.
• unexpected inputs be used as a test method to determine the robustness of a
software application.
• By designing tests that provide invalid, unexpected, or random data as inputs to
the software, one can observe how the application handles such inputs.
• This method helps in identifying unhandled exceptions, crashes, and potential
security vulnerabilities like buffer overflows and injection flaws.
• For example, if a system is designed to accept numerical input values between 1 and 10, a robustness test
would involve trying to input values outside of this range, such as 0, 11, or -5, to see how the system responds

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 7

 Obfuscation techniques to evade detection
• Obfuscation means to make something difficult to understand.
• Obfuscation is an umbrella term for a variety of processes that transform
data into another form in order to protect sensitive information or personal
data.
• Three of the most common techniques used to obfuscate data are encryption,
tokenization, and data masking.
• Malware authors may use techniques such as packing, encryption, and
polymorphism to obscure malicious code.
• Countermeasures include using advanced static analysis tools that can
deobfuscate code, and employing heuristic and behavior-based detection
systems that do not rely solely on signatures.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 8

 Ethical Implications of reverse engineering software
• This issue is largely debated and does not seem to have a clear cut
answer. Ethically, reverse engineering can be a contentious issue.
• The number one argument against reverse engineering is that of intellectual
property. If an individual or an organization produces a product or idea, is it
ok for others to "disassemble" the product in order to discover the inner
workings?
• It is often justified for finding and fixing security vulnerabilities, ensuring
interoperability, or validating intellectual property rights.
• However, it can also lead to copyright infringement, privacy violations, and
competitive harm if not bounded by legal frameworks and ethical
guidelines.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 9

 Methods to secure software
• Methods to secure software against reverse engineering attacks
• Developers can use a range of techniques to protect against reverse
engineering include using code obfuscation, implementing secure coding
practices to minimize vulnerabilities, employing anti-tamper mechanisms,
• Techniques also using cryptographic signatures to protect code integrity,
and utilizing hardware-based security solutions like TPM or HSM.
• TPM(Trusted Platform Modules) stores keys securely within your device,
while HSM(Hardware Security Modules) offers dedicated hardware for
key storage, management, backup, and separation of access control.
• Anti-reverse engineering methods help protect mobile apps from being
reverse-engineered.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 10

 Challenges of reverse engineering
• Challenges of reverse engineering in different programming languages
• Higher-level languages like Java or C# that compile to intermediate
representations (like bytecode) can be easier to reverse engineer than lower-
level languages like C/C++ because tools can reconstruct a form closer to the
original source code.
• In contrast, compiled binary code of lower-level languages is less readable
and requires more effort to understand.
• But C is the mostly high-level programming language used in reverse
engineering. As the C/C++ compiler is available for most of the OS platforms,
its programs can be run on all standard platforms. All these nature of C/C++
made this one of the good programming languages for hackers and crackers.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 11

 Guideline for safely testing malware
• Guidelines should include using isolated environments (sandboxes),
ensuring all network connections are controlled or disabled to prevent the
spread of malware, employing up-to-date antivirus tools to scan and
monitor activities, and maintaining strict access controls.
• To prevent malware attacks,
1.Patch and update software.
2.Use firewalls and security software, such as antimalware and antivirus.
3.Follow email security best practices.
4.Deploy email security gateways.
5.Avoid clicking links and downloading attachments.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 12

 Methods for detecting and defending against
code injection attacks
• Code injection is the term used to describe attacks that inject code into an
application.
• That injected code is then interpreted by the application, changing the way a
program executes.
• Code injection attacks typically exploit an application vulnerability that
allows the processing of invalid data.
• Detection can involve analyzing code for improper input validation and data
handling practices.
• Defense strategies include implementing comprehensive input sanitation and
validation routines, employing least privilege principles, and using security
frameworks that inherently protect against such attacks.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 13

 Role of reverse engineering
• Mostly in the lifecycle of software development and maintenance.
• Reverse engineering can aid in understanding legacy code, recovering lost
documentation, debugging, and interoperability.
• In maintenance, it helps identify parts of the code that need refactoring or security
updates.
1.Cost Reduction: In some cases, reverse engineering can be a cost-effective
alternative to creating products or components from scratch.
2.Repairing existing products. Backwards engineering can help engineers repair
and bend existing products.
3.Competitive Analysis: In business, reverse engineering allows companies to
analyze competitors' products and gain a competitive edge.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 14

 Legal and Ethical hacking practices
• reverse engineering can help in legal and ethical hacking practices.
• To begin with, ethical hacking is authorized and aims to improve security.
In contrast, malicious hacking is both illegal and harmful.
• Ethical hackers have permission to test systems, while malicious hackers
exploit vulnerabilities for personal gain or damage.
• In ethical hacking, reverse engineering aids in vulnerability assessments
and penetration testing by understanding how applications handle security
mechanisms.
• Legally, it can help audit compliance with security standards and
intellectual property use.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 15

 Tools like IDA Pro and OllyDbg
• Reverse engineering tools like IDA Pro and OllyDbg enhance software
security analysis.
• These tools help in disassembling binaries to human-readable assembly
code, debugging applications to understand runtime behavior, and
analyzing executable without source code. This analysis is crucial for
uncovering hidden vulnerabilities and ensuring application robustness.
• IDA is primarily geared towards static analysis (though it does have
dynamic debugging functionality)
• OllyDbg (and x32dbg/x64dbg, Immunity Debugger, WinDbg, Cheat
Engine, etc.) is geared towards dynamic analysis.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 16

 Analyze worms and viruses
• A Worm is a form of malware that replicates itself and can spread to different
computers via Network. A Virus is a malicious executable code attached to
another executable file which can be harmless or can modify or delete data.
• method to handle and analyze worms and viruses found during a reverse
engineering process.
• Start by isolating the malware in a secure, controlled environment. Use
disassembly and debugging tools to analyze the code and understand its
functionality and propagation mechanism. Document the behavior and
develop signatures or heuristic patterns for detection. Finally, contribute
findings to security databases and communities to help others in detection and
mitigation.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 17

 Unit 5:- Software Reverse Engineering
• Software Reverse Engineering:
• Defend against software targets for viruses,
• worms and other malware,
• improving third party software library,
• identifying hostile codes-buffer overflow,
• provision of unexpected inputs.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 18

 Unit 6:- Computer crime and Legal issues
• Computer crime and Legal issues: Intellectual property.
• privacy issues.
• Criminal Justice system for forensic.
• audit/investigative.
• situations and digital crime procedure/standards for extraction,
preservation, and deposition of legal evidence in a court of law.

DEPARTMENT OF COMPUTER ENGINEERING, Sanjivani COE, Kopargaon 19