2020 Nss Labs Aep Product Rating Report Check Point Sandblast Agent PDF
2020 Nss Labs Aep Product Rating Report Check Point Sandblast Agent PDF
Handcrafted Attacks
Testing was based on the Advanced Endpoint Protection Test Methodology v4.0 (available at www.nsslabs.com)
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
Table of Contents
Security........................................................................................................................................................................................... 3
Tuning and False Positives............................................................................................................................................................................... 3
Resistance to Evasions..................................................................................................................................................................................... 3
Malware Delivered over Email ........................................................................................................................................................................ 4
Malware Delivered over HTTP ........................................................................................................................................................................ 4
Drive-by Exploits .............................................................................................................................................................................................. 5
Social Exploits .................................................................................................................................................................................................. 6
Handcrafted (Targeted) Attacks ...................................................................................................................................................................... 6
Management & Reporting Capabilities ............................................................................................................................................ 7
Authentication ................................................................................................................................................................................................. 7
Policy ................................................................................................................................................................................................................ 7
Logging ............................................................................................................................................................................................................. 7
Change Control ................................................................................................................................................................................................ 7
Alert Handling .................................................................................................................................................................................................. 8
Reporting ......................................................................................................................................................................................................... 8
Total Cost of Ownership (TCO) ........................................................................................................................................................ 9
3-Year Total Cost of Ownership ...................................................................................................................................................................... 9
Test Environment .......................................................................................................................................................................... 10
Appendix ...................................................................................................................................................................................... 11
Authors ......................................................................................................................................................................................... 12
Test Methodology ......................................................................................................................................................................... 12
Contact Information...................................................................................................................................................................... 12
Table of Figures
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
2
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
Security
The threat landscape is evolving constantly; attackers are refining their strategies and increasing both the volume and complexity of
their attacks. Enterprises now are having to defend against everyday cybercriminal attacks as well as targeted attacks and even the
rare advanced persistent threats (APTs). For this reason, we tested using multiple commercial, open-source, and proprietary tools to
employ attack methods that are currently being used by cybercriminals and other threat actors. We increased the levels of difficulty
as we tested, beginning with common attacks, escalating to targeted attacks, and then applying obfuscation techniques to see if we
could evade defenses. We then recorded whether the endpoint protection blocked and logged threats accurately and how frequently
it triggered false positives.
Blocked Detected Rating
Tuning and False Positives 2/645 (0.3%) 3/645 (0.5%) AA
This test includes a varied sample of legitimate application traffic that Detected
may be falsely identified as malicious (also known as false positives). As 3
0.5%
part of the initial setup, we tuned the endpoint protection as it would
be by a customer. Every effort was made to eliminate false positives
while achieving optimal security effectiveness and performance, as Download
Blocked
would be the aim of a typical customer deploying the device in a live - FALSE POSITIVES
network environment. To ensure that the vendor did not deploy 0.0%
unrealistic (overly aggressive) security policies that blocked access to Execution Not Blocked
legitimate software and websites, we tested the endpoint protection Blocked or Detected
2 640
against 645 false positive samples, including but not limited to the 99.2%
0.3%
following file formats: .exe, .jar, .xls, .xlsm, .accdb, .css, .pdf, .doc, .docx,
.zip, .DLL, .js, xls, .chm, .rar, .Ink, .cur, .xrc., .slk, .ppt, pptx, .iqy, .htm. Figure 1 – False Positives
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
3
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
One of the most common ways in which users are compromised is Not Blocked
through malware delivered over email. For several years, the use of or Detected
-
social engineering has accounted for the bulk of cyberattacks against 0.0%
consumers and enterprises. Socially engineered malware attacks
Detected
often use a dynamic combination of social media, hijacked email -
0.0% EMAIL MALWARE
accounts, false notification of computer problems, and other
deceptions to encourage users to download malware. One well- Execution
Download
Blocked
known social engineering attack method is spear phishing. Blocked
1,459
72
Cybercriminals use hijacked email accounts to take advantage of the 4.7%
95.3%
implicit trust between contacts and deceive victims into believing that
the sender is trustworthy. The victim is tricked into opening the email
attachment, which then launches the malicious malware program. Figure 3 – Malware Delivered over Email
To test how well the endpoint protection is able to protect against this type of attack, malware was emailed to the user. The desktop
client then retrieved the email and opened/executed the malware. If the malware was blocked, the corresponding time was recorded.
We deployed a CentOS 7.7.1908 Linux mail store with kernel 3.10.0-957.5.1.el7.x86_64 running Dovecot v2.2.36 for IMAP as the mail
server. Victim machines consisted of a combination of 32-bit and 64-bit Windows 7 endpoints and 64-bit Windows 10 endpoints.
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
4
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
While there are millions (or hundreds of millions) of malware samples Execution
Blocked
in circulation at any given point in time, they are frequently delivered -
by exploits that target consumer desktops known as drive-by exploits. 0.0%
While vulnerabilities are patched and defenses against exploits incorporated into new versions of operating systems (i.e.,
Windows), many organizations cannot easily upgrade due to financial, technical, or other constraints. As of January 2020,
NetMarketShare1 reports OS market share for Windows 7 (released 11 years ago in 2009) at 25.56% and for Windows 10
(released in 2015) at 57.08%.
Research has shown that oftentimes the most valuable assets have the most stringent change control to avoid business
interruption. This creates a challenging dynamic whereby the most valuable assets tend to be the most difficult to defend
(e.g., older OS, unpatched, etc.). Therefore, as vulnerabilities are patched and defenses against exploits are incorporated into
new versions of operating systems (i.e., Windows)—which makes exploitation of computers more difficult—the value of
endpoint protection is often associated with its ability to protect older, unpatched, and generally more vulnerable systems.
1 https://netmarketshare.com
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
5
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
provides attackers with a wide attack surface. As such, sending social Figure 6 – Social Exploits
exploits through mass email (phishing), could yield profit as the number of
victims would be large, albeit smaller than in the case of malware since exploits would have technical dependencies.
To test how well the product was able to protect against social exploits, we deployed 19 victim machines. All of the machines were
running Windows 10 version 1709 (OS Build 16299.15). Machines were configured with Internet Explorer 11 (version 11.15.16299.0 –
Update Version 11.0.47) and Microsoft Office 2016 (version 16.0.7431.2032).
The aim of this test was to see which endpoint products were able
to protect customers while under adverse conditions dictated by
the attacker. In this case, we wanted to find out which products
could block new handcrafted (unknown) malware while being Not Blocked
or Detected
prevented from accessing cloud services. Detected 13
- 61.9%
What happens, for example, if an employee goes on a business 0.0%
trip to China where Internet traffic is tight ly controlled? In such a Execution HANDCRAFTED ATTACKS
scenario, access to the corporate VPN is likely blocked and the Blocked
4
security software on the employee’s laptop may not be able to 19.0% Download
receive updates or communicate in general. What happens if the Blocked
4
employee’s laptop is attacked with targeted malware? 19.0%
For the purposes of this test, handcrafted (targeted) malware was
created by modifying the source code of keyloggers, ransomware, and Figure 7 – Handcrafted (Targeted) Attacks
destructoware, and then recompiling the binary so that it was new to the
products being tested. We then attempted to infect a host (e.g., a laptop) with the malware and recorded whether or not the
endpoint protection blocked the attack.
Because creating samples in this manner is a painstaking and time-consuming exercise, we tested only a handful of targeted samples;
results should be viewed with this in mind.
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
6
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
Central management is available with a thick or thin client; we used both clients for testing.
AA
Authentication Logging
Role-based access control (RBAC) that allows for several Signatures corresponding to malicious traffic from endpoints
defined roles for administrative users, such as “Admin,” are logged and displayed centrally. The logs of the CMS
“Analyst,” “Reporting User,” “Event User,” etc. is supported. contain administrative actions, including session login/logout,
The system supports third-party authentication via static successful authentication, and unsuccessful authentication
password, OS password, RADIUS, SecurID, TACACS, and LDAP. attempts. Additionally, policy changes and users that modified
the policy are tracked. (which user deployed to what
Policy groups/machines, etc.) is included in the logs. For the purpose
of forensic analysis, the endpoint agent captures all events.
Management enables administrators to create and save
Changes made during an attack, and logs from multiple
multiple security policies, with no pre-defined limit. Bulk
sources are correlated using a common time zone.
operations enable a single operation to be applied to multiple
groups. The policy mechanism is diverse and supports all sorts
of use cases, enabling true customization. The system is able
to add custom rules, white-list, and black-list to build a custom
policy that can be applied to machines, users, and groups of
machines/users. Inheritance (nested rules) is fully supported,
including creation of groups and sub-groups such that sub-
groups can inherit certain aspects of configuration and policy
definition from parent groups. Versioning enables
administrators to view the policy version of each endpoint.
Change Control
To prevent tampering, the policy is encrypted both in transit
The management system provides a change control/audit log
and in storage. The encryption mechanism has a built-in
that contains the username, date and time of the changes, as
verification to validate the policy integrity. The policy storage
well as details of the change. Neither revision history nor
is protected by the product’s self-protection mechanism
policy rollback is supported.
(kernel level) to avoid tampering that is native to the
Sandblast Agent’s protection.
When an alert is selected, the system provides the ability to
directly access and view the policy that triggered the event.
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
7
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
8
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
Expected Costs for Check Point Software Technologies SandBlast Agent – 2,500 Agents
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
9
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
Test Environment
• BaitNET™ (NSS Labs Proprietary)
• 32-bit Microsoft Windows 7 (Version 6.1 (Build 7601: SP1)
• 64-bit Microsoft Windows 7 (Version 6.1 (Build 7601: SP1)
• 64-bit Microsoft Windows 10 (version 1607 (Build: 14393.0)
• 64-bit Microsoft Windows 10 (version 1709 (Build: 16299.15)
• Adobe Acrobat Reader 19.021.20061
• Adobe Flash Player 18.0.0.160
• Adobe Flash Player 32.0.0.207
• Adobe Flash Player 32.0.0.223
• Adobe Flash Player 32.0.0.238
• Adobe Reader 9.40
• Adobe Reader DC 2017.012.20093
• Google Chrome 78.0.3904.70
• Kali (Kernel release 4.19.0-kali1-amd64)
• Microsoft Internet Explorer 9.0.8112.16421
• Microsoft Internet Explorer 10.0.9200.16438
• Microsoft Internet Explorer 11.0.14393.0
• Microsoft Office Professional 2013 version 15.0.5119.1000 (Microsoft Word, Excel, PowerPoint, Access, etc.)
• Microsoft Office Professional 2016 version 16.0.7341.2032 (Microsoft Word, Excel, PowerPoint, Access, etc.)
• Microsoft Silverlight 5.1.20125
• Microsoft Silverlight 5.1.50918
• Oracle Java 6 Update 27
• Oracle Java 8 Update 181
• Oracle Java 8 Update 211
• Oracle Java 8 Update 221
• Oracle Java 8 Update 231
• Rapid7 Metasploit (v5.0.46-dev)
• VMware vCenter (Version 6.7u2 Build 6.7.0.30000)
• VMware vSphere (Version 6.7.0.30000)
• VMware ESXi (Version 6.7u3 Build 14320388)
• Wireshark version 3.0.3
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
10
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
Appendix
RATING DEFINITION
A product rated ‘AAA’ has the highest rating assigned by NSS Labs. The product’s capacity to meet its
AAA
commitments to consumers is extremely strong.
A product rated ‘AA’ differs from the highest-rated products only to a small degree. The product’s capacity to
AA
meet its commitments to consumers is very strong.
A product rated ‘A’ is somewhat more susceptible to sophisticated attacks than higher-rated categories.
A
However, the product’s capacity to meet its commitments to consumers is still strong.
A product rated ‘BBB’ exhibits adequate protection parameters. However, sophisticated or previously unseen
BBB
attacks are more likely to negatively impact the product’s capacity to meet its commitments to consumers.
A product rated ‘BB,’ ‘B,’ ‘CCC,’ ‘CC,’ and ‘C’ is regarded as having significant risk characteristics. ‘BB’ indicates
the least degree of risk and ‘C’ the highest. While such products will likely have some specialized capability and
protective characteristics, these may be outweighed by large uncertainties or major exposure to adverse
conditions.
A product rated ‘BB’ is less susceptible to allowing a compromise than products that have received higher-risk
BB ratings. However, the product faces major technical limitations, which could be exposed by threats that would
lead to its inability to meet its commitments to consumers.
A product rated ‘B’ is more susceptible to allowing a compromise than products rated ‘BB’; however, it
B currently has the capacity to meet its commitments to consumers. Adverse threat conditions will likely expose
the product’s technical limitations and expose its inability to meet its commitments to consumers.
A product rated ‘CCC’ is currently susceptible to allowing a compromise and is dependent upon favorable
CCC threat conditions for it to meet its commitments to consumers. In the event of adverse threat conditions, the
product is not likely to have the capacity to meet its commitments to consumers.
A product rated ‘CC’ is currently highly susceptible to allowing a compromise. The ‘CC’ rating is used when a
CC failure has not yet occurred but NSS Labs considers a breach a virtual certainty, regardless of the anticipated
time to breach.
A product rated ‘C’ is currently highly susceptible to allowing a compromise. The product is expected to fail to
C
prevent a breach and to not have useful forensic information compared with products that are rated higher.
A product rated ‘D’ is actively being breached by known threats and is unable to protect consumers. For non-
specialized products, the ‘D’ rating category is used when protecting a consumer is unattainable without a
major technical overhaul. Unless NSS Labs believes that such technical fixes will be made within a stated grace
D
period (often 30-90 calendar days), the ‘D’ rating also is an indicator that it is a virtual certainty that existing
customers using the product have already experienced a breach—whether they know it or not—and should
take immediate action.
This report is Confidential and is expressly limited to NSS Labs’ licensed clients.
11
CHECK POINT SOFTWARE TECHNOLOGIES SANDBLAST AGENT V81.20.7425 ENDPOINT PROTECTION
Authors
Rabin Bhattarai, Thomas Skybakmoen, Vikram Phatak
Test Methodology
NSS Labs Advanced Endpoint Protection (AEP) Test Methodology v4.0 is available at www.nsslabs.com.
Contact Information
NSS Labs, Inc.
3711 South Mopac Expressway
Building 1, Suite 400
Austin, TX 78746
[email protected]
www.nsslabs.com
This and other related documents are available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs.
© 2020 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system, e-mailed
or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. (“us” or “we”).
Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these conditions, you
should not read the rest of this report but should instead return the report immediately to us. “You” or “your” means the person who accesses
this report and any entity on whose behalf he/she has obtained this report.
1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it.
2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All use of and
reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of any nature whatsoever
arising from any error or omission in this report.
3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO
EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSS
OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the
hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or
that the products will meet your expectations, requirements, needs, or specifications, or that they will operate without interruption.
5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report.
6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective
owners.
This and other related documents are available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs.
© 2020 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system, e-mailed
or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. (“us” or “we”).