Key findings and conclusions:

With 364,000 TCP connections per second, Cisco ASA 5585-X handled 102% more connections per second than Juniper

Lab Testing Summary Report

April 2011
Report 110419

Cisco throughput with EMIX frames reached 24.5 Gbps an 11% increase compared to the SRX3600 ASA 5585-X can sustain 10 million concurrent connections At maximum load, Cisco used 425 watts, while Juniper used 1168 watts at idle, a 64% difference in power consumption

Product Category:

Enterprise Firewall

Vendors and Products Tested:

isco engaged Miercom to evaluate the performance of the ASA 5585-X SSP-60 Adaptive Security Appliance. The ASA 5585-X was tested in a variety of scenarios to determine the maximum TCP and UDP throughput performance. Parameters recorded included CPU utilization, allocated memory utilization, connections per second (CPS), concurrent connections, real world HTTP throughput, and TCP EMIX traffic. We performed the identical tests on a Juniper SRX3600 Services Gateway to compare and contrast the performance of these products. In addition, we also measured the power consumption of each appliance while under load. The Cisco ASA 5585-X SSP-60 has a multi-core, multi-processor architecture. The tested model featured twenty-four processing cores, six Gigabit Ethernet interfaces, and four 10 Gigabit Ethernet interfaces. The appliance combines a stateful firewall and VPN capabilities in one device, and includes features such as Layer 2 and Layer 3 firewall operation, advanced inspection engines, IPSec VPN, SSL VPN, and clientless SSL VPN.

Cisco ASA 5585-X Juniper SRX3600

Figure 1: Cisco ASA 5585-X and Juniper SRX3600 TCP EMIX Traffic
25.0 24.5 24.0 23.5

Gbps

23.0 22.5 22.0 21.5 21.0 20.5 20.0

Cisco ASA 5585-X 24.5 Juniper SRX3600 22.0

Source: Miercom, April 2011

Gbps

Cisco ASA 5585-X achieved 24.5Gbps throughput for TCP EMIX traffic, an 11% improvement attained by ASA 5585-X when compared to the Juniper SRX3600.

How We Did It
To fully exercise the performance of the products, the test bed utilized BreakingPoint and Spirent TestCenter products. Bidirectional test traffic was generated using BreakingPoint version: 2.1.0.0 build number: 71254 strikebuild: 78528, and the Spirent Test Center v3.5.5. Real-world HTTP tests were performed using HTTP v1.1 with persistence while transferring objects of varying sizes. TCP performance tests were conducted using BreakingPoint to generate 64-byte HTTP traffic, as well as EMIX traffic containing a mix of packet sizes and protocols. UDP performance tests utilized Spirent TestCenter to send fixed frame sizes ranging from 64-byte up to 9,216-byte jumbo frames (9,192 bytes on Juniper). The Cisco ASA 5585 SSP-60 was equipped with four 10GE interfaces. Cisco Adaptive Security Appliance (ASA) Software v8.4.1 and Cisco Security Manager (CSM) 4.1 were used during testing. The product architecture features a multi-processor/multi-core platform with 24 processing cores. Default MTU size for TCP traffic was 1,380 bytes to allow for overhead. Default MTU size for UDP traffic was 9,216 bytes. Juniper SRX3600 was configured with four 10GE interfaces and JunOS 10.4r2.6. Most recent publicly available documentationfor the product states it as providing up to 30 Gbps of firewall performance and 175,000 connections per second. Default MTU size for TCP traffic was 1,460 bytes. Default MTU size for UDP traffic was 9,192 bytes. The SRX3600 has a NPU-based architecture with XLR variants featuring 2-8 cores per SPC. The tests in this report are intended to be reproducible for customers who wish to recreate them with the appropriate test and measurement equipment. Current or prospective customers interested in repeating these results may contact [email protected] for details on the configurations applied to the Device Under Test and test tools used in this evaluation. Miercom recommends customers conduct their own needs analysis study and test specifically for the expected environment for product deployment before making a product selection.

Tested Configurations
Platform Operating System Product Architecture Processing Cores Gigabit Ethernet Interfaces 10 Gigabit Ethernet Interfaces Cisco ASA 5585-X SSP-60 ASA v8.4.1 and CSM 4.1 Multi-processor, multi-core 24 0 4 Juniper SRX3600 JunOS 10.4r2.6 NPU based with XLR variants 2-8 cores per SPC 0 4

Test Bed Diagrams

Cisco ASA 5585 SSP-60

ASA 5585 SSP-60 UDP Topology

(All interfaces and connections are 10 GbE SR)

Cisco ASA 5585 SSP-60

BreakingPoint

ASA 5585 SSP-60 TCP Topology

(All traffic generation is 10 GbE SR)

Spirent TestCenter

Copyright 2011 Miercom

Cisco ASA 5585 and Juniper SRX3600

Page 2

EMIX - Real World Protocol Mix

To further evaluate the performance of each appliance, a mix of packet sizes and protocols were used. We constructed a mixed traffic profile that reflects a more realistic representation of a typical network. Each protocol was assigned a specific weighting, with a preponderance of traffic being HTTP, as this is most representative of an enterprise environment. Protocol % Bandwidth Allocated HTTP 43.956 BitTorrent Peer 21.978 IMAP v4 Advanced 16.484 FTP 8.791 SMTP 8.791 We observed a 24.6 Gbps throughput for this mix of traffic on the ASA 5585, 11% higher than the throughput for the SRX3600. Juniper obtained a maximum throughput of 22 Gbps. These results can be seen graphically in Figure 1 on page 1.

Concurrent Connections
The objective of this test is to determine the maximum number of concurrent or simultaneous TCP connections that the firewall can handle. The sessions are simulated using 64-byte HTTP packets, and all sessions are kept open once established and increased until the maximum upper limit is reached, as reported by the firewall itself. CPU and memory utilization is not relevant for this test and was not recorded. The ASA5585-X SSP-60 achieved 100% of its expected value, establishing a maximum of 10 million concurrent connections. The Juniper data sheet states an upper limit of 2.25 million concurrent connections for the SRX3600. In our testing, the SRX3600 exceeded that target, establishing a maximum of 2.39 million connections.

Figure 2: Maximum Concurrent TCP Connections using 64-byte Frames

10

Millions of Connections

8 6 4 2 0
Cisco ASA 5585 10.0 Juniper SRX3600 2.39

319% higher!

Concurrent Connections In Millions

Source: Miercom, April 2011

The maximum number connections is shown.

of

concurrent

HTTP Maximum Throughput

To understand how well each firewall processed HTTP traffic, we created a scenario using web traffic of varying packet sizes. We configured our test equipment to deliver an average packet size of 471 bytes by selecting an object size of 11,110 bytes and changing the TCP Maximum Segment Size (TCPMSS) to 1,140. This created a varying packet size which is more process intensive. We recorded the maximum throughput achieved for each appliance without incurring packet loss.

Figure 3: Maximum Throughput

17.4 17.3 17.2 17.1 17.0 16.9 16.8 16.7 16.6 16.5 16.4

The ASA 5585-X SSP-60 delivered 3.5% more throughput than the Juniper, achieving 17.3 Gbps with no packet loss. Resource utilization reporting showed that the CPU was nearly maxed out at 99%, while memory utilization was 21%. Juniper achieved 16.7 Gbps with no packet loss. Resource reporting indicated that the CPU was only 11% utilized and memory only 40% utilized. As noted in the previous test, we feel that this number is too low considering the stress the appliance was under, and suspect the resource allocation is being reported incorrectly.

Gbps

Gbps

Cisco ASA 5585 17.3

Juniper SRX3600 16.7

Source: Miercom, April 2011

Copyright 2011 Miercom

Cisco ASA 5585 and Juniper SRX3600

Page 3

Connections per Second

In this test, our objective was to determine the maximum number of new TCP connections each firewall could handle without dropping any packets. The TCP sessions were simulated using 64-byte HTTP packets. The connection rate was ramped up until a maximum number of connections was achieved. CPU and memory utilization was recorded at this point. The ASA 5585-X SSP-60 achieved a 49% increase in performance over the Juniper SRX3600, handling a maximum new connection rate of 364,000 connections per second. Resource utilization of 100% CPU and 22% memory was recorded for the ASA 5585. The Juniper SRX3600 was able to handle 180,000 new TCP connections per second. We noted that the Juniper appeared to incorrectly report its resource utilization, as the CPU usage was reported to be only 11% and memory was reported at 40%. As the unit could not handle more than 180K connections without incurring packet loss, it seemed unlikely that the CPU was being so lightly stressed. This anomaly has been observed in other Juniper testing. See Figure 4.
400,000 350,000 300,000 250,000

Figure 4: Connections per Second

CPS

200,000 150,000 100,000 50,000

Cisco ASA 5585-X Max CPS 364,000 Source: Miercom, April 2011

Juniper SRX3600 180,000

UDP Mixed Packet Sizes with Jumbo Frames

To determine the maximum data rate each appliance could sustain with no packet loss for a range of fixed packet sizes, including jumbo frames, the standard RFC 2544 Benchmarking Throughput test was used. 8,000 hosts were used to provide enough traffic to maximize the throughput potential for each packet size. The maximum jumbo frame size for the ASA 5585-X is 9,216 bytes. The maximum jumbo frame size for the SRX3600 is 9,192 bytes. We also evaluated the throughput of an IMIX traffic stream consisting of a mix of various packet sizes. As seen in Figure 5, the Cisco ASA 5585 outperformed the Juniper SRX3600 at every frame size. We noted that the throughput for the Juniper appeared to trail off at higher frame sizes. In Figure 6 on the next page, it can be seen that the Cisco ASA 5585 was able to handle a higher packets per second rate than the Juniper SRX3600 for all packet sizes.

Figure 5: UDP Mixed Packet Sizes

Maximum Data Rate Sustained with No Packet Loss
45 40

Throughput (Gbps)

35 30 25 20 15 10 5 0

35.3 25.5 18.2 15.2 12.1 6.7 4.6 8.2

37.7

39.8

39.8

23.4

23.6

23.6

23.6

23.6

22.2 21.0

64
Source: Miercom, April 2011

128

256

512

1024

1280

1518

Jumbo Frames

IMIX

Cisco ASA 5585

Juniper SRX3600

ASA 5585 outperformed the SRX3600 at every frame size. Throughput decreased slightly for the Juniper as frame sizes increased.

Copyright 2011 Miercom

Cisco ASA 5585 and Juniper SRX3600

Page 4

Figure 6: Maximum Data Rate Sustained with No Packet Loss

12,000 10,000

iPackets per Second in Thousands

9,920.7
8,000 6,000 4,000 2,000 0

10,172.2 8,223.7 6,830.6 6,906.1 6,852.2 5,980.9 5,482.5 4,223.0 2,824.7 3,613.5 2,268.6 1280 3,230.0 1,917.2 1518 538.1 323.3 IMIX
Juniper SRX3600

7,257.5

6,859.3

64

128

256

512

1024

Jumbo Frames
Cisco ASA5585

Source: Miercom, April 2011

Frame Size

ASA 5585 outperformed the SRX3600 in every frame size in the packets per second test.

Management
Cisco Security Manager (CSM) is the enterprise class security management solution that enables enterprises to manage and scale security operations efficiently. This powerful graphical management solution enables consistent policy enforcement, quick troubleshooting of security events, and summarized reports from across the security deployment (see Figure 7). While enterprise customers can leverage CSM for large scale management, Cisco Adaptive Security Device Manager (ASDM) can be used for managing smaller sized networks. Cisco ASDM is included with all Adaptive Security Appliances and the product can be used to quickly configure, monitor and troubleshoot ASA firewalls.

Figure 7: Cisco Security Manager

Management interface screen shows a large variety of firewall event views available to an administrator. Device IDs, source and destination IP addresses, and service type are clearly displayed for analysis. Filters can be created based on any field, not just by event type.

Power Consumption
We conducted a power consumption evaluation between the two security devices. We used the standard RFC 2544 Benchmarking Throughput test script for 100% traffic load. Each device had two power supplies, a firewall module installed and no IPS. ASA 5585-X used 382 watts at idle and 425 watts at full load. The Juniper SRX3600 at idle had recorded 1,168 watts and 1,249 watts for 100% load. Juniper used 194% more power at maximum load. These tests were run several times in order to be certain the figures were accurately recorded. This is a dramatic advantage for the Cisco security appliance. Copyright 2011 Miercom

Figure 8: Power Consumption at Idle and Maximum Load (EMIX)

1400 1200 1000

Watts

800 600 400 200

Cisco Watts

Juniper

Cisco

Juniper

ASA 5585 SRX3600

ASA 5585 SRX3600

382 1168 IDLE STATE

425 1249 MAXIMUM LOAD

Source: Miercom, April 2011

*Lower is better

Cisco ASA 5585 and Juniper SRX3600

Page 5

Miercom Performance Verified

Based on its lab testing of the Cisco ASA 5585-X SSP-60 Adaptive Security Appliance, Miercom verifies that the throughput capabilities of this security appliance are superior to that of the Juniper SRX3600 Services Gateway. Hands on testing results confirmed the ASA 5585 sustained 10,000,000 simultaneous connections, 364,000 connections per second, and HTTP traffic at 17.3 Gbps. Cisco had better performance at all packet sizes, including jumbo frames. The Cisco ASA 5585-X SSP-60 delivers impressively on the security, scalability and performance required for enterprise networks, data centers and Web 2.0 applications. Performance and security features earned the Cisco ASA 5585-X the Miercom Performance Verified Certification.

Cisco ASA 5585

Cisco Systems Inc. 170 West Tasman Drive San Jose, CA 95134 www.cisco.com 1-800-553-6387

About Miercoms Product Testing Services

Miercom has hundreds of product-comparison analyses published over the years in leading network trade periodicals including Network World, Business Communications Review - NoJitter, Communications News, xchange, Internet Telephony and other leading publications. Miercoms reputation as the leading, independentproducttestcenter is unquestioned. Miercoms private test services include competitive product analyses, as well as individual product evaluations. Miercom features comprehensive certification and test programs including: Certified Interoperable, Certified Reliable, Certified Secure and Certified Green. Products may also be evaluated under the NetWORKS As Advertised program, the industrys most thorough and trusted assessment for product usability and performance.

Report 110419

[email protected]

Before printing, please consider electronic distribution

Product names or services mentioned in this report are registered trademarks of their respective owners. Miercom makes every effort to ensure that information contained within our reports is accurate and complete, but is not liable for any errors, inaccuracies or omissions. Miercom is not liable for damages arising out of or related to the information contained within this report. Consult with professional services such as Miercom Consulting for specific customer needs analysis.

Copyright 2011 Miercom

Cisco ASA 5585 and Juniper SRX3600

Page 6