Secure One Services

Support Partner Troubleshooting Guide

How to Monitor & Manage:

PROXYSG, PROXYAV, Reporter, Director

JANUARY 2014 Confidential - For Internal and Official Partner Use Only
 USER GUIDE
Table of Contents
Introduction 4 Connectivity Issues 12 Collecting Data for HTTPS Syslog 25
Issues 18 Enabling Syslog Monitoring 25
BlueTouch Support Partner Possible Causes for
SSL proxy debug 18 Health Monitoring 25
Troubleshooting Tools 4 Connectivity Issues 12
CFssl debug 19 Changing Threshold and
Proxy Network and Interface
BlueTouch Online 4 Settings 12 HTTP Debug 19 Notification Properties: 25
Blue Coat Knowledge Base 4 Collecting Data for Adaptor FTP Issues 19 Modify the notification
Blue Coat Forums 4 issues 13 settings. 26
Collecting Data for FTP
Bridging 13 SNMP 26
Support Engagement Process 5 Issues 19
Collecting Data for Bridging Configuring SNMP 26
ProxySG Basics 6 Instant Messaging Issues 20
Issues 14 Obtaining MIB Files 26
How the ProxySG Works 6 Network Infrastructure 14 Collecting Data for IM
Proxy AV Basics 27
Issues 20
Troubleshooting Methodology 7 Comprehensive PING Test 14
How the Proxy AV Works 27
WCCP Issues 15 Streaming Issues 20
Important Note Concerning Data Content Scanning Modes 27
Proxy Hangs and Crashes 15 Collecting Data for Streaming ProxyAV Deployment Types 28
Gathering: 7 Issues 20
Hang 15 General Data Gathering
Optimized Setup 7 Hardware 21 Guidelines 28
Crash 16
General Data Gathering Authentication Issues 16 FRU Items 21 ProxyAV Monitoring 31
Guidelines 8 Non FRU Items 21
Collecting Data for Director 32
Restart Issues 9 Authentication Issues 16 CRU 21
How Director Works 32
Overview 9 BCAAA Debug 16 Initial Visual Inspection 21
Gathering Troubleshooting
Collecting Data for Restart How to start the BCAAA Diagnosing Boot Sequence (for Data. 33
Issues 9 debug 16 510/810 SG and 810 AV models
ONLY) 22 Director Monitoring 34
How to stop the BCAAA
High CPU Conditions 10 Disk Drive, Power Supply, and
debug 17 Reporter 35
Possible Causes for High CPU Fan issues 22
Enabling Windows SSO How Reporter Works 35
Conditions 10 Debug 17 RMA 22 Gathering Troubleshooting
Collecting Data for High CPU Disabling the Windows SSO
Advance Hardware Data. 35
Conditions 10 Debug 17
Replacement 22 Memory Related Issues 35
Memory Pressure Conditions 11 Gathering the BCAAA and
RMA Cut-Off Times 23 Reporter Crash issues 35
Windows SSO Debug Logs 17
Collecting Data for Memory ProxySG Monitoring 23 Appendix A: Opening Service
Pressure Conditions 11 HTTP Issues 17
Requests 38
Collecting Data for HTTP Event Logging 23
Slowness Issues 11 Setting Event Log Level 23 BlueTouch Online 38
Issues 17
Possible Causes for Slowness HTTP Debug 18 Setting Event Log Size 23 Creating a New SR on BTO 38
SUPPORT PARTNER Issues 11 Email Alerts 24
HTTPS Issues 18 Telephone 40
TROUBLESHOOTING GUIDE Collecting Data for Slowness Enabling Event
Issues 11 Notifications: 24 Americas: 40

2
 USER GUIDE
Latin America: 40 Forcing a Core 49 Appendix L: Director Messages
Mexico: 40 Core Location/Retrieval 50 File (Audit Logging) 62
Europe, Middle East and Files to Upload 50 Appendix M: Reporter
Africa: 40 Full Core with Packet Capture Diagnostics File 63
Asia-Pacific: 40 Included 51
Updating a Service Request 40 Quick Reference 51 Appendix N: Reporter Journal
Definition of Terms 51 File 64
Appendix B: Manually Retrieving
Files from the ProxySG 42 Appendix G: ProxySG How
To… 52
Appendix C: Uploading Files to
Blue Coat from ProxySG 43 Archive and Restore the
ProxySG Configuration 52
ProxySG Management Console
GUI (MC) 43 Backing up the
Configuration 52
upload.bluecoat.com 43
Restoring the
ftp.bluecoat.com 43 Configuration 52
Uploading Files Using the Errors 52
CLI 43
Archive Using the CLI 53
Appendix D: How to Create a
Restoring the Configuration
Policy Trace on the ProxySG 45 File Using the CLI 53
Default Policy Trace 45 Export-Import SSL Keys 54
Tracing requests from one Export the SSL key 55
Client IP address 45 Import the SSL Key 55
Saving the policy trace 46 Reinitializing the Disk(s) on the
Appendix E: How to Create ProxySG 56
a Packet Capture on the Single Disk System 56
ProxySG 47 Multiple Disk System 56
Overview 47 Factory Restoring the
Device 57
Limitations 47
Methods 47 Appendix H: ProxyAV Logs.zip
Management Console 47 File 58
Browser URL 48 Appendix I: How to create
Filters 48 a Packet Capture on the
Appendix F: Core Files on the ProxyAV 59
ProxySG 49 Appendix J: Taking a Packet
Overview 49 Capture on Director 60
SUPPORT PARTNER Basic Configuration 49 Appendix K: Taking a Debug
TROUBLESHOOTING GUIDE Core Generation 49 Dump on Director 61

3
 USER GUIDE
Introduction
This document is intended to assist in properly configuring your proxy
device to have the highest probability of capturing the appropriate
troubleshooting data in the event that a problem should occur.
Proper configuration of the device ahead of time and gathering the
appropriate data in a timely manner increases the probability of
determining root cause of a problem. The outlined proactive and reactive
procedures are key in minimizing business impacting downtime.
This document will guide you through the recommended configuration Service Management
and provides step-by-step procedures for collecting data in the event You can view current/existing service requests, as well as creating
you should experience problems with the device. new ones.
The screenshots and commands listed are based on the current Long Checking on the status of RMA’s, show a list of Products owned along
Term Recommended version of SGOS, which is 6.2.x. with Entitlements.
If commands differ between this and other versions of SGOS these have Service Requests | Open New SR
been included and highlighted.
Blue Coat Knowledge Base
BlueTouch Support Partner Troubleshooting Tools https://kb.bluecoat.com

There are a number of varying types of troubleshooting tools available to Access the latest solutions and research technical issues in our product-
partners. They are listed below. specific knowledgebase.
Can be accessed using your BTO credentials.
BlueTouch Online
https://bto.bluecoat.com Has up-to-date and accurate information regarding Technical Field
Alerts , Security Advisories , Known Issues & Solutions even FAQs,
BlueTouch Online is the portal for all your support needs including; covering ALL Blue Coat Products.
• Open Service Requests
Blue Coat Forums
• Access the Knowledgebase
https://forums.bluecoat.com
• Access Forums
Collaborate with peers and subject matter utilizing experts to answer
• Read Tech Briefs your support questions. Share knowledge and news about Blue Coat
products and related technologies on the new Blue Coat Forums. You
• View Field Alerts and Security Advisories
must register for privileged partner access, please ensure to use the
• Downloading the latest SGOS versions and SNMP MIB files same email address you used for your main BTO account.
• Licensing
SUPPORT PARTNER • Training Documentation
TROUBLESHOOTING GUIDE

4
 USER GUIDE
Support Engagement Process
Alternatively for P1 & P2 Issues you can call
Blue Coat Support to log a new Ticket, rather
than doing so via BTO.
*However please ensure that all necessary
debug data has been collected before calling.

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

5
 USER GUIDE
ProxySG Basics It’s important to understand that the ProxySG intercepts traffic.
That means it will terminate the client connection and create a new
COMMON TERMS connection to the OCS through which it will make requests on behalf
SG Blue Coat ProxySG of the client. The proxy acts as the server to the client and acts as the
OCS Origin Content Server (web server serving the actual content) client to the server. It is the middleman.
MC GUI Management Console (https://x.x.x.x: 8082)
Deployment Types
CLI Command Line Interface (accessed via serial cable or SSH)

CPL Content Policy Language Explicit:

(used for writing policy directly into local or central policy files) The client is configured to make requests directly to the proxy IP
VPM Virtual Policy Manager (GUI for creating policy launched via “Configuration > Policy > address. In this deployment the client is aware that it is talking to a
Visual Policy Manager”.
proxy and will behave accordingly. Either via explicit Proxy settings in
PCAP Packet Capture the Client Browser, or via a PAC File
X.X.X.X Variable indicating to place your SG’s IP address here
Transparent:
The client is unaware of the proxy’s presence and believes that it
How the ProxySG Works
is talking directly to the OCS. Transparency can be accomplished
The ProxySG as a forward proxy sits at the edge of the network and by deploying the proxy “in-line” (in-path) or via WCCP by layer 2-4
acts as a gateway between the LAN and the internet. The proxy can be redirection.
deployed as an explicit proxy or a transparent proxy, however, no matter
how it is deployed its basic function remains the same and that is to
act as an intermediary between devices on the LAN and web servers
located on the internet as well as to secure, control and accelerate
that traffic.

Request Request

Response ProxySG Response

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

6
 USER GUIDE
Troubleshooting Methodology 4. Open an SR and/or contact support
• Use BlueTouch Online to open an SR to include a maximum of
1. Clear understanding of the issue
details.
• Details, Details, Details!
• If the issue is critical then use the instructions for direct telephone
• When did the issue start?
access to support to raise the priority of the created SR.
• How often does it occur?
• Is this a new or existing setup and was it ever working?
Important Note Concerning Data Gathering:
• What changed?
As a forward proxy is essentially the networks gateway to the internet
›› Network changes (routers, firewall, new switch, etc).
it will often have a large amount of traffic passing through it especially
›› Deployment (added subnet, changed to explicit, changed PAC
during peak production hours. In order to effectively gather data useful
file, etc).
for analysis and debugging the greatest effort should be made to
›› Policy changes, configuration changes, etc. limit the amount of noise (unrelated data) captured in PCAP’s,
›› OCS changes (have content/headers on server changed?) traces, and logs where possible.
• How “exactly” does the end user experience the problem? The SG’s logging and capturing features have size limitations that if
›› Provide URLs and screen shots of errors where possible. reached could render the data useless as it will either stop logging/
capturing or the data will begin to wrap and overwrite previously
• How many users are impacted?
captured data. For example a busy proxy on a sizable network has been
• What applications are involved? known to fill its PCAP buffer in as little as 6 seconds and a 6 second
• What devices are involved? PCAP is doubtful to contain any useful information. It’s therefore advised
• Precise, detailed, step-by-step, duplication steps. to filter pcaps where possible and only run for the duration of replication.
(PCAP Filters can be found in Appendix E)
• What is the impact to production?
2. Check the knowledgebase and forums for answers
Optimized Setup
• Access BlueTouch Online: https://bto.bluecoat.com/support
• Select “Knowledge Base” The Blue Coat ProxySG ships with a default configuration for capturing
troubleshooting data such as logs and snapshots that can be made
• Alternatively browse to https://kb.bluecoat.com
more useful via a few optimizations. Every SG in your organization
3. Gather the necessary data should run the following basic setup. This setup will be more effective
• Data gathering guidelines for most issues are detailed in this in capturing the data we need to troubleshoot issues the first time the
document. problem happens.
• Verify the optimized setup is in place. Configuring the ProxySG parameters
• Find the issue that relates (same or similar) and follow the 1. Maintenance > Core Image > full
instructions for gathering the proper data.
a. Apply
• If your issue is not covered here follow the “General Data Gathering
Guidelines” (located just after the instructions for “Optimized 2. Snapshots
SUPPORT PARTNER
Setup” below). a. Maintenance -> Service Information -> Snapshots
TROUBLESHOOTING GUIDE

7
 USER GUIDE
i. Click “New” General Data Gathering Guidelines
ii. Enter name: “CPU<proxyserialnumber>”
This document is intended to assist in gathering the right data, the right
iii. Highlight the newly created snapshot and click “Edit” way, the first time. However, it cannot possibly cover every possible
1. target: /Diagnostics/CPU_Monitor/Statistics/Advanced scenario. In the event the issue is not specifically covered here or when
2. interval: 5 (minutes) in doubt use these instructions for gathering data in conjunction with the
3. Maximum number to store: 100 optimized setup.
4. Check “Enabled” and click “OK”
1. Start a policy trace limited to a single IP (see Appendix D)
5. Apply
2. Start a packet capture (see Appendix E)
iv. Click “New”
v. Enter name: “sysinfo_stats5” 3. Reproduce the issue as simply and concisely as possible.
vi. Highlight the newly created snapshot and click “Edit” 4. Stop the packet capture
1. target: /sysinfo-stats
5. Save the policy trace to your PC using the browsers FILE|SAVE
2. interval: 5 (minutes)
function and save as TEXT.
3. Maximum number to store: 100
4. Check “Enabled” and click “OK” 6. Open an SR (see Appendix A)
5. Apply
7. Upload the following (see Appendix C)
vii. Modify the existing snapshot_sysinfo_stats to store 100 • Sysinfo
snapshots • Event log
1. Highlight the sysinfo_stats snapshot and click “Edit”
• Snapshots (all)
2. Leave the interval the same.
3. Maximum number to store: 100 • Packet Capture
4. Click “OK” • Policy trace (https://upload.bluecoat.com)
5. Apply
b. Verify that the packet capture filter on the SG is empty.
(Maintenance > Service Information > Packet Captures)
The following change should ONLY be implemented when high
CPU issues are being encountered on the ProxySG device.
1. CPU Monitor
a. Management Console
i. Statistics>Advanced>Diagnostics > Start the CPU Monitor
Once enabled it will write the CPU Monitor output to the sysinfo
and snapshot files each time they are written (configured via the
“interval” option in the snapshots configuration settings.
SUPPORT PARTNER
TROUBLESHOOTING GUIDE

8
 USER GUIDE
Table 1 below provides a matrix to assist you in knowing what data to capture based on the issue you are troubleshooting.

PACKET
CAPTURE CONTEXT ACCESS EVENT POLICY COMMAND NETWORK
PROBLEM SYSINFO (PROXY) SNAPSHOT OR CORE LOG LOG TRACE OUTPUT DIAGRAM
URL X X X X X

RESTART X X X X

REPORTER X X X

WCCP X X X X

ICAP OR AV X X X X X

IM X X X

STREAMING X X X

BRIDGING X X X X

AUTHENTICATION
X X
(BCAAA)

PERFORMANCE/
X X X X X
CONNECTIVITY

POLICY ISSUE X X X

SSL X X

MACH 5 X X X X X

Restart Issues Configuring the ProxySG parameters

Overview • Full cores dumps everything from memory into a file at the time of the
Restarts are often able to be solved by Blue Coat code changes, crash, which means that it takes longer to write than context cores do.
but sometimes restarts can be caused by web servers using poorly In some cases all we need is a context, but there are many cases
written applications or other requests as well as high CPU or high when we need a full core. A full core is always preferred to a context
memory pressure. In the majority of cases the cause of a restart will core for proper analysis, however if you don’t feel you can afford the
be determined via the analysis of a core file (context, memory, or full downtime it takes to capture a full core then you have the option
core), however it is always vital to gather the other logs and snapshots of changing the setting to write a “context only”. It’s important to
available on the SG as well. remember, however, that if we can’t get the information we need from
the context then we will have to set the box up to write a full core
For more information concerning cores please refer to Appendix F: and wait for the box to crash a second time in order to gather the
Cores information we need.

Collecting Data for Restart Issues Gathering Data

If the optimized setup has been configured then your SG is already • The SG will crash and come back up on its own. Before it comes back
configured properly to write the necessary data we need for restart up the SG will write the core and other files to disk. During the time
SUPPORT PARTNER issues. the SG is writing a core it will be inaccessible. Full cores can take
TROUBLESHOOTING GUIDE 5-15 minutes to complete depending on the amount of memory, type
of hardware, etc.
9
 USER GUIDE
Uploading Data to Blue Coat Support 3. Policy is highly complex or contains a large amount of REGEX
resulting in resource intensive processing.
Upload the following information
4. Bug.
• Sysinfo
• Event Log Collecting Data for High CPU Conditions
• Snapshots (all 4 - sysinfo, syinfo_stats, cpu<serial#>, sysinfo_stats5) 1. Configuring the ProxySG parameters
• Enable CPU Monitor (Statistics>Advanced>Diagnostics > Start the
• PCAP (saved to core file, see appendix: Cores)
CPU Monitor)
• Full Core and/or Context Core • Ensure Snapshot sysinfo_stats interval is set to at least “5” minutes
in order to catch the spike
High CPU Conditions 2. Gathering Data
Possible Causes for High CPU Conditions • Wait for CPU Spike
1. Traffic • Start a PCAP and let run for 30 seconds.
• The SG is undersized for the amount of traffic it is required to • Stop and download the PCAP to your PC.
process. • Save the sysinfo to your PC (https://x.x.x.x:8082/sysinfo)
• The SG is getting hammered by malicious traffic (virus or poor • Force a full core (whilst the spike is still occurring)
application behavior). ›› enter the CLI
• The SG has been exposed to the internet and is functioning as an ›› type “restart abrupt”
“open proxy”.
›› Wait for the SG to come back on its own…it will take a little while
• The network is looping requests causing a race condition on (5-15 min).
the SG.
3. Data to upload
• Performance issues with authentication agent causing bottleneck
on the SG. • Upload the following information
›› sysinfo
2. Lack of available resources to process data.
›› event log
• Content Filtering Memory Allocation is set incorrectly requiring
the use of disk to accomplish policy application tasks thereby ›› snapshots (all 4 - sysinfo, syinfo_stats, cpu<serial#>, sysinfo_
increasing the load on the CPU. stats5)

• The SG is experiencing communication issues with an ICAP server ›› PCAP

or the ICAP server is having issues processing requests which ›› full core and context (if able to generate during the CPU spike)
will cause the SG to queue scanning requests thereby decreasing
available resources and increasing load on the CPU.
• The SG is sending inappropriate data to the ICAP server for
scanning such as streaming data.
SUPPORT PARTNER • Other network related communication problems are causing
TROUBLESHOOTING GUIDE requests to begin queuing due to slowness thereby increasing load
on the CPU.

10
 USER GUIDE
Memory Pressure Conditions **Important**
Memory pressure is similar to economics law of supply and demand The Sysinfo and snapshot data must be downloaded from the ProxySG
in that it is a measurement of SG’s ability to free (supply) and allocate device BEFORE it is rebooted or a Full Core forced, as the Threshold
(demand) memory to requesting processes. If the demand outpaces Monitor statistics written to both is erased when the device is rebooted.
the supply then memory pressure will rise. It is normal for memory As a reboot clears out the Memory on the device along with these
pressure to fluctuate, however in cases where demand outpaces supply memory related statistics, which are important to see which components
perpetually the SG will eventually run out of resources resulting in are consuming memory at the time the spikes/increase have occurred.
slowness, hangs, and in some cases restarts.
Memory pressure levels can be monitored via SNMP. The SG also allows Slowness Issues
warning thresholds to be set (Maintenance > Health Monitoring) and the
event log will log “TCP regulation memory pressure” type messages in Possible Causes for Slowness Issues
the event the SG may be experiencing problems with memory pressure. Slowness can be caused by many different issues or a combination of
issues. Some things to look at…
Collecting Data for Memory Pressure Conditions
• Network problems (dropped packets, routing, firewalls, etc.)
1. Configuring the ProxySG parameters
• In most cases the optimized setup will suffice. • Speed/duplex mismatch

2. Gathering Data • DNS (slow response from server, failures, etc.)

• Save the sysinfo to your PC (https://x.x.x.x:8082/sysinfo) • Authentication (large auth policy, not using auth caching, slow auth
• Save the snapshots to your PC (https://x.x.x.x:8082/Diagnostics/ return, etc)
Snapshot) • Off-box services (AV slow processing or problems communicating,
• Force a full core Content Filtering)
i. enter the CLI • Circuit capacity and speed (Is the pipe saturated?)
ii. type “restart abrupt”
• High Memory pressure
iii. wait for the SG to come back on it’s own…it will take a little while
(10-15 min). • High CPU

3. Upload via the MC • Undersized device for environment

• sysinfo Collecting Data for Slowness Issues

• event log 1. Configuring the ProxySG parameters
• snapshots (all 4 - sysinfo, syinfo_stats, cpu<serial#>, sysinfo_ • In most cases the optimized setup will suffice in conjunction with a
stats5) packet capture.
• full core and context
2. Gathering Data
• The data required to analyze a slowness issue may vary depending
SUPPORT PARTNER on whether high CPU, high memory pressure, or authentication
TROUBLESHOOTING GUIDE issues are involved. For example, if the SG is slow and CPU is
high, then follow the instructions for gathering data for high CPU as

11
 USER GUIDE
it’s possible that the slowness may only be a symptom of the real 6. Transparent Redirection (i.e. WCCP)
problem (that which is causing high CPU). An authentication related
7. Proxy Hang
slowness issue might require a BCAAA debug in conjunction with
the packet captures you will take. With any slowness issue packet 8. Proxy Crash
captures are required.
Proxy Network and Interface Settings
i. Start a packet capture on the client and the SG (2 captures are
then running simultaneously). In order for the ProxySG to communicate effectively it must have an
active network adaptor (NIC) and properly configured network settings.
ii. Duplicate the slowness issue.
iii. Stop the PCAP on the SG. 1. Verify the ProxySG adaptor settings
iv. Stop the PCAP on the client. a. Configuration > Network > Adapters
b. Select the correct interface and check IP address and subnet mask
3. Upload via the MC
c. Click “Interface Settings”
• Sysinfo
i. Verify speed/duplex
• Event log
ii. Verify “Allow transparent interception” on the internal interface.
• Snapshots (all 4 - sysinfo, syinfo_stats, cpu<serial#>, sysinfo_
(DO NOT enable transparent interception on public (internet)
stats5)
facing interfaces unless the SG is being used as a reverse proxy.)
• Packet capture from the SG
2. Verify the adaptor is active
• Client PCAP (upload via https://upload.bluecoat.com)
a. Ping the IP address bound to the interface from the CLI on the SG
*If the SG PCAP doesn’t show anything but slowness occurring, itself
sometimes an unfiltered pcap at the time may help identify if the device
b. Check the link light on the adaptor itself
is being overrun with packets, or if indeed there is something else in play
which is causing the issue (ARP, DNS etc etc) c. Verify the cable is plugged into the proper adaptor
d. Verify the cable is good (swap out for known good cable)

Connectivity Issues
Connectivity issues can be tricky to diagnose at first as there may be a
number of things actually happening. Sometimes what is first thought to
be a failure to communicate to the SG actually turns out to be something
unrelated to network communications.

Possible Causes for Connectivity Issues

1. Physical interface on the Proxy
2. Network settings on the Proxy
3. Bad Ethernet cable
SUPPORT PARTNER 4. Bridging
TROUBLESHOOTING GUIDE 5. Network infrastructure (router/switch)

12
 USER GUIDE
Collecting Data for Adaptor issues Configuration > Network > Adaptors > Bridges
1. Sysinfo
2. Event log
3. Snapshots (all)
4. Packet capture taken during the issue
5. Output from: https://x.x.x.x:8082/TCP/Statistics

Bridging
The ProxySG provides bridging functionality by two methods:
• Software: A software, or dynamic, bridge is constructed using a set
of installed interfaces. Within each logical bridge, interfaces can be
assigned or removed.
• Hardware: A hardware, or pass-through, bridge uses a 10/100 dual
interface Ethernet adapter. This type of bridge provides pass-through
support.
A pass-through adapter is a 10/100/1000 dual interface Ethernet adapter
which provides an efficient fault-tolerant bridging solution. If this adapter
is installed on a ProxySG, SGOS detects the adapter on system boot
and automatically creates a bridge – the two Ethernet interfaces serve
See “Chapter 68: Software and Hardware Bridges” in the SGOS
as the bridge ports. If the ProxySG is powered down or loses power
Administration Guide for the respective version of SGOS for more
for any reason, the bridge fails open; that is, traffic passes from one
information on bridges.
Ethernet interface to the other.
When running in a failover configuration (as below),
Check to see if the bridge is created and the correct interfaces are
associated with it: Master

Slave

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

13
 USER GUIDE
“Enable Spanning Tree” must be checked on the bridge interface in Network Infrastructure
order to avoid bridging loops. (see figure below) A common mistake in diagnosing connectivity issues is to discount
the network infrastructure that lies between the workstation and the
ProxySG. Any one of these devices could suffer from hardware or
configuration issues resulting in packet loss, blocking, misroutes, etc.
The first step in diagnosing a connectivity issue is to run a ping from
the workstation to the ProxySG. If the ping fails the next step is to run a
more comprehensive ping test that includes the network infrastructure
making up the physical path between the workstation and ProxySG to
find out exactly where the failure might be.
Comprehensive PING Test

192.168.1.1 192.168.2.1 192.168.2.3 65.102.24.35

192.168.1.2 192.168.2.2
207.45.26.12

Ping test performed from the 192.168.1.2 workstation.

1. ping 192.168.1.2 (device itself)
2. ping 192.168.1.1 (gateway router for the 192.168.1.0 network)
Collecting Data for Bridging Issues
3. ping 192.168.2.1 (router interface 192.168.2.0 net)
The following information is required for bridging issues…
4. ping 192.168.2.2 (SG interface)
1. Sysinfo
5. ping 192.168.2.3 (firewall 192.168.2.0 internal net interface)
2. Event log
6. ping 65.102.24.35 (firewall external interface)
3. Snapshots (all)
7. ping 207.45.26.12 (web server)
4. Packet capture
In this example the proxy is deployed “in-line” so it is bridging (layer 2
5. Output from the following locations:
function). If ping is successful to the firewalls internal interface then the
a. Via the console: proxy has connectivity and is passing traffic (at least on a network level).
i. https://x.x.x.x:8082/Bridge/fwtable One thing to remember is that ping uses ICMP and if any of the devices
ii. https://x.x.x.x:8082/Bridge/stats along the physical path is configured to block ICMP then ping tests will
not be beneficial. Telneting to one of the service ports on the SG (port 80
b. Via the command line interface:
or 443 for example) might be a better test, but often packet captures are
i. show bridge fwtable <bridge name> required to see what is actually happening.
SUPPORT PARTNER ii. show bridge conf <bridge name>
TROUBLESHOOTING GUIDE iii. show bridge statistics <bridge name>

14
 USER GUIDE
WCCP Issues Proxy Hangs and Crashes
The following information is required for WCCP issues: Sometimes the SG may appear hung when the real problem may be
an issue with connectivity. Likewise what may at first appear to be
1. Sysinfo a connectivity issue is in reality an SGOS hang or crash. If the proxy
2. Event log is hung, it will not respond to ping or serial access. If the proxy has
crashed and is writing a full core, for example, it will not respond to ping
3. Snapshots (all) until it has restarted itself. This condition might last anywhere from 2-20
4. Packet capture from the SG taken while the issue is occurring. minutes depending.

5. Network Diagram In order to better determine the actual state of the SG it is important to
know the answer to the following (before rebooting the box):
6. Router logs: output from the following commands when issued on the
router (when WCCP is active): Is the device accessible via the following methods?
• sho ver • Ping
• sho conf
• Telnet
• sho ip wccp
• Serial Cable
• show ip wccp <service-group> detail
• show ip wccp <service-group> view • HTTP
• display WCCP events • HTTPS GUI
Gathering data Failure to access the SG via HTTP/HTTPS console, SSH, or telnet from
1. Start the PCAP on the SG (unfiltered…if a filter exists delete, apply, a workstation might indicate some sort of a connectivity issue (whether
and start the PCAP) with the proxy or the network), but if the SG cannot be accessed via
serial cable then most likely it is hung or has crashed and is restarting or
2. Reproduce the issue writing a core.
3. Stop the PCAP Hang
Upload via the MC 1. Force a core
• Sysinfo a. Serial Session: ctrl-x ctrl-h
• Event log b. DO THIS ONLY ONCE!! It may seem like nothing is happening, but
• Snapshots (all) it may be dumping the core. If ctrl-x ctrl-h is entered a second time
it will overwrite the first core and the dump will be useless.
• Packet Capture
c. If after 20 min the SG has not restarted on it’s own then proceed to
• Zip the router logs together and upload via https://upload.bluecoat.com step #2 below.
2. Reboot the SG and upload the following
a. Sysinfo
SUPPORT PARTNER b. Event log
TROUBLESHOOTING GUIDE c. Snapshots (all)
d. Core (if one was written)
15
 USER GUIDE
Crash a. If using Windows Single Sign On (WinSSO) start an SSO debug also
(instructions below).
Check to see if the SG has crashed by looking at the “Core Image”
section of the SYSINFO. 3. Start a policy trace
1. Using the browser enter: http://x.x.x.x:8082/cm/core_image/ 4. Start a packet trace on the SG
details?All
5. Reproduce the issue
2. Check the latest minicontext date and time to see if it correlates with
6. Stop the PCAP
the time when connectivity was lost.
7. Save the policy trace (refresh and use FILE | SAVE and save as TEXT)
Example:
Minicontext produced on: 2009-02-13 07:30:31+00:00UTC 8. Zip up the entire BCAAA directory (this contains the debug and
configuration files)
3. If the SG has crashed follow the instructions in this document
concerning “Restart Issues”. 9. Upload
a. Sysinfo
Authentication Issues b. Event log
c. Snapshots (all)
The following information is required for any authentication related issue:
d. Packet Capture
1. Sysinfo
e. Zip file containing the BCAAA directory and policy trace (upload via
2. Event Log https://upload.bluecoat.com)
3. Snapshots (all) 10. Post the details and IP address information to the case.
4. Packet Capture
BCAAA Debug
5. Policy Trace (limited to a single IP) The BCAAA debug is enabled via a modification of the bcaaa.ini file
6. bcaaa.ini file located in the BCAAA installation directory (default: C:\Program Files\
Blue Coat Systems\BCAAA).The debug logs are saved to the BCAAA
7. BCAAA debug (Windows SSO debug if applicable) installation directory in the same place the bcaaa.ini file is located and
8. Authentication method (NTLM, Radius, LDAP, etc.) are named “BCAAA-nnn.log” with “nnn” representing the PID for the
BCAAA process that created the log. Each process generates a separate
9. Authentication server type and OS version.
log so there may be several log files created at the end of the debugging
10. Details of the issue and IP address information of involved devices. session.

Collecting Data for Authentication Issues How to start the BCAAA debug

1. Log the affected (or test) user off from the ProxySG 1. Open bcaaa.ini

This can be done via the advance URL https://x.x.x.x:8082/Auth/User- 2. Append the following to the end of the file
Logins/Summary/Realm/ [Debug]
SUPPORT PARTNER
2. Start the BCAAA debug (instructions below) on the server to which the DebugLevel=0xFFFFFFFF
TROUBLESHOOTING GUIDE
SG is configured to make authentication requests. This server should 3. Save the bcaaa.ini
be running the BCAAA.
16
 USER GUIDE
4. Restart the BCAAA service 2. Restart the BCAAA service
5. Debug logs are written to the BCAAA installation directory Gathering the BCAAA and Windows SSO Debug Logs
How to stop the BCAAA debug Zip up the entire BCAAA installation directory (C:\Program Files\Blue
Coat Systems\BCAAA by default). This will contain:
1. Remark out the 2 lines added to the bcaaa.ini with a semi-colon (;)
1. bcaaa.ini
;[Debug]
;DebugLevel=0xFFFFFFFF 2. bcaaa debug logs
2. Restart the BCAAA service 3. sso.ini
4. sso debug logs
5. debug_dcq_primary_full.sso and debug_dcq_primary_inc.sso (located
in the “%ProgramFiles%\Blue Coat Systems\BCAAA” directory)

HTTP Issues
The following information is required for HTTP issues (anything that uses
HTTP such as URL requests, FTP over HTTP, Reverse Proxy, etc.).
1. Sysinfo
2. Event Log
3. Packet Capture
Enabling Windows SSO Debug
4. Policy Trace (limited to a single workstation IP)
1. Open the sso.ini file found in the BCAAA install directory (default is C:\
5. HTTP debug
Program Files\Blue Coat Systems\BCAAA)
6. Details of the URL and any log-on details (on the website in question)
2. Add the following under the [DCQSetup] heading
or steps to reproduce the problem (step-by-step). Include screenshots
a. DCQDebug=1
if necessary.

Collecting Data for HTTP Issues

1. Start the policy trace
2. Start the HTTP debug (instructions below)
3. Start a PCAP on the SG
3. Restart the BCAAA service 4. Reproduce the issue
Disabling the Windows SSO Debug 5. Stop the PCAP
SUPPORT PARTNER
TROUBLESHOOTING GUIDE 1. Remark out the “DCQDebug=1” entry from the sso.ini file using a 6. Save the HTTP debug (refresh and use FILE | SAVE and save as TEXT)
semi-colon (;)
7. Save the policy trace (refresh and use FILE | SAVE and save as TEXT)
17
 USER GUIDE
8. Upload via the MC 8. Details of the URL and any log-on details or steps to reproduce the
• Sysinfo problem (step-by-step).
• Event log
Collecting Data for HTTPS Issues
• Snapshots (all)
1. Start a Policy Trace (limited to single IP)
• Packet Capture
2. Start a packet capture on the SG
• Zip file containing the policy trace and HTTP debug log (upload via
https://upload.bluecoat.com) 3. Start the SSL and HTTP debug logs (instructions below)

9. Post the details and IP address information to the case. 4. Reproduce the issue

HTTP Debug 5. Stop the packet capture

• https://x.x.x.x:8082/HTTP/debug_http_all 6. Stop and save the policy traces and debug logs (refresh and use FILE
| SAVE and save as TEXT)
• Clear log (“Clear http debug all level”)
7. Zip up the policy trace and debug logs
• Click “Set Level to Debug”
8. Upload via MC
• Click SUBMIT
• Sysinfo
• Display HTTP Debug Info
• Event log
• Refresh page to view new content • Snapshots (all)
• Packet capture
HTTPS Issues • Zip file containing policy trace and debug logs (upload via
This section assumes that HTTPS is being intercepted (HTTPS service is https://upload.bluecoat.com)
enabled and an SSL intercept policy layer is installed). It’s important to 9. Post the details and IP address information to the case.
keep in mind that HTTPS traffic is encrypted. While packet captures are
SSL proxy debug
helpful to a point they do not allow any visibility into the data itself. In
order to see into the data we need to use SSL and HTTP debug logs. • https://x.x.x.x:8082/SSLproxy/DEBUG
The following information is required for HTTPS issues: • Clear log
1. Sysinfo • Set debug mask
2. Event Log • Add all available mask values
3. Packet Capture SSL debug mask should look like this:
4. Snapshots (all) Current mask value is:
5. Policy Trace (limited to a single IP) SSLPROXYWARN +
SSLPROXYERROR +
6. SSL Debug (SSL Proxy for mainly Explicit intercepted traffic and CFssl
SUPPORT PARTNER SSLPROXYNOTICE +
for some explicit handling and all transparently intercepted SSL traffic)
TROUBLESHOOTING GUIDE SSLPROXYINFO
7. HTTP debug

18
 USER GUIDE
• Display ssl proxy debug info FTP Issues
• Refresh Page to display output after test has been ran The following information is required for FTP issues:
• Save the debug log output (Use File|Save and save as text) 1. Sysinfo
• To revert back, remove “SSLPROXYINFO” from the current mask 2. Event Log
by using the “Remove” link next to this value under “Change Mask
3. Snapshots (all)
Value:” on the “Set Debug Mask” page.
4. Packet capture
CFssl debug
5. Policy Trace (limited to single IP address)
• https://x.x.x.x:8082/cfssl/debug
6. FTP client version and settings
• Clear log
7. Details such as IP address of the workstation and URL of the FTP site
• Set debug mask
8. Authentication information (used for replication and identification of
• Add all available mask values
user in the PCAP)
CFSSL debug mask should look like this:
Collecting Data for FTP Issues
Current mask value is:
1. Start the policy trace
CFSSLWARN +
CFSSLERROR + 2. Start a PCAP on the SG
CFSSLNOTICE + 3. Reproduce the issue
CFSSLINFO
4. Stop the PCAP
• Display ssl proxy debug info
5. Save the policy trace (refresh and use FILE | SAVE and save as TEXT)
• Refresh Page to display output after test has been ran
6. Upload via the MC
• Save the debug log (Use File|Save and save as text) • Sysinfo
• To revert back, remove “CFSSLINFO” from the current mask by using • Event log
the “Remove” link next to this value under “Change Mask Value:” on • Snapshots (all)
the “Set Debug Mask” page
• Packet Capture
HTTP Debug • Zip file containing the policy trace (upload via
• https://x.x.x.x:8082/HTTP/debug_http_all https://upload.bluecoat.com)

• Clear log (“Clear http debug all level”) 7. Post the details of the issue, FTP client version and settings, FTP
URL, authentication information, and client IP address to the case.
• Click “Set Level to Debug”
• Click SUBMIT
SUPPORT PARTNER • Display HTTP Debug Info
TROUBLESHOOTING GUIDE
• Refresh page to view new content

19
 USER GUIDE
Instant Messaging Issues 2. Event Log

The following information is required for instant messaging issues… 3. Packet captures from the client and the SG

1. Sysinfo 4. Policy Trace (limited to a single IP address)

2. Event Log 5. Access logs (streaming)

3. Packet captures from the client and the SG 6. Client type, version, and network settings.

4. Policy Trace (limited to a single workstation IP) 7. Steps to reproduce the problem (detailed step-by-step).

5. Messenger client type, version, and network settings. 8. Details and IP information of involved devices.

6. Steps to reproduce the problem (detailed step-by-step). Collecting Data for Streaming Issues
7. Details and IP information of involved devices. 1. Enable Access Logging
• Configuration > Access Logging
Collecting Data for IM Issues
• Check “Enable Access Logging
1. Start a policy trace (Limited to single IP address)
• Apply
2. Start a PCAP on the client
2. Start policy trace
3. Start a PCAP on the SG
3. Start packet capture on the client
4. Reproduce the issue
4. Start the packet capture on the SG
5. Stop the PCAP’s
5. Reproduce the issue
6. Save the policy trace (refresh and use FILE | SAVE and save as TEXT)
6. Stop the PCAP
7. Zip the policy trace
7. Save the policy trace (refresh and use FILE | SAVE and save as TEXT)
8. Upload via the MC
8. Zip up the client PCAP and policy trace
• Sysinfo
9. Upload via the MC
• Event log
• Sysinfo
• Snapshots (all)
• Event log
• PCAP from the SG
• Snapshots (all)
• Zip file containing the policy trace and client PCAP (upload via
https://upload.bluecoat.com) • PCAP from the SG
• Access logs (streaming)
9. Post details and IP information to the case.
• Zip file containing the client PCAP and policy trace (upload via
https://upload.bluecoat.com)
Streaming Issues
10. Post details and IP information to the case.
SUPPORT PARTNER The following information is required for streaming issues…
TROUBLESHOOTING GUIDE
1. Sysinfo

20
 USER GUIDE
Hardware CRU
Hardware replacements occur in two forms. Any faults identified with the components listed above (Non FRU items)
will result in a CRU RMA being required.
1. FRU (Field Replacement Unit)
A CRU will be shipped which will include the Main Chasis of the device
2. CRU (Chasis Replacement Unit) and the components listed above.

FRU Items ** IMPORTANT – Upon placing a CRU RMA ALL original FRU items must
1. Hard Drive be REMOVED from the old chassis to insert into the new chasis when
received.
2. Add-on Cards
This includes the Hard Disk, Power Supply, System Fan (if applicable),
3. Power supplies System Blower (if applicable) and any additional Add-on Cards.
4. System Fan
Initial Visual Inspection
See the table below to determine which FRU apply to which supported
HW model

HARD POWER SYSTEM SYSTEM ADD-ON

DISK SUPPLY FAN BLOWER CARDS
SG510 Y Y N Y N

SG810 Y Y N Y N

SG300 Y Y N N N*

SG600 Y N Y N N*

SG900 Y Y Y N N*

SG9000 Y Y Y N N

AV810 Y Y N Y N/A

AV1200 Y Y Y N N/A

AV1400 Y Y Y N N/A

AV2400 Y Y Y N N/A

*NIC and SSL cards On-Board in this Model

The following information is required for initial inspection
**Correct as of Jan 2014
1. Sysinfo
Non FRU Items
2. Event Log
1. Motherboard
3. Serial console output/errors
2. CPU
3. Memory
SUPPORT PARTNER 4. Front Panel Display
TROUBLESHOOTING GUIDE
5. Other HW Components (including Compact Flash Card)

21
 USER GUIDE
Diagnosing Boot Sequence RMA
(for 510/810 SG and 810 AV models ONLY)
Blue Coat provides RMA Advanced Hardware Exchanges to those
These hardware models rely on the hard Disk drives to store the customers who have a valid entitlement under product warranty or
configuration and SGOS images which the device accesses during the service contract. Below are the steps to take to initiate a RMA.
boot sequence.
1. Open a technical support case via BlueTouch Online or contact
An issue with the primary disk in these models can cause issues with Technical Support.
devices not booting correctly.
2. A technical support engineer will work with you to troubleshoot the
issue and verify if a hardware repair or replacement is required.
3. Upon confirmation that an RMA is required (either an FRU or CRU) the
technical support engineer will re-assign the ticket to the Blue Coat
Customer Care team.
4. The Blue Coat Customer Care team will initiate the RMA by obtaining
the following customer information:
• Company Name
• Shipping Address
• Contact Name
• Contact Phone Number
• Contact email address
• Problem Description
• Product Model Number
• Product Serial Number
The following information is required for diagnosing the boot sequence
5. Only upon receipt of the information requested above can the Blue
1. Serial Console output Coat Customer Care team place the RMA order.
2. Troubleshooting steps taken 6. When the RMA has shipped, the customer will receive a shipment
3. Sysinfo (if bootable) notification which will include instructions regarding the defective
hardware return.
4. Event Log (if bootable)
7. If the defective hardware is not returned in a timely manner, the
Disk Drive, Power Supply, and Fan issues customer will be contacted. Customers may also contact Blue Coat
The following information is required for disk drive, power supply and for return instructions, [email protected].
fan issues
Advance Hardware Replacement
1. Sysinfo
RMA Requests received and deemed necessary by Technical Support
SUPPORT PARTNER 2. Event Log before the RMA cut off time will have replacement hardware shipped
TROUBLESHOOTING GUIDE same day. Requests received or verified by Blue Coat Technical Support
after the RMA cut off time ship the following day. Actual delivery time
22
 USER GUIDE
will vary dependent upon shipping origin and destination. Out-of-box Event Logging
warranty shipments may require additional time to ship. The SG can be configured to log system events as they occur. Event
logging allows you to specify the types of system events logged, the
RMA Cut-Off Times
size of the event log, and to configure Syslog monitoring. The appliance
(Daylight Savings Time observance may affect RMA cut-off times where can also notify you by e-mail if an event is logged.
applicable)
Setting Event Log Level
SUPPORT REGULAR
CENTER BUSINESS HOURS RMA CUT OFF TIME 1. Select Maintenance > Event Logging > Level.
NORTH AMERICA Mon-Fri, 06:00 to 18:00, Mon-Fri 12:00, Sat 10:00, Sun *10:00,
Pacific Time Zone Pacific Time Zone

EUROPE Mon-Fri, 08:00 to 17:00, GMT Mon-Fri 11:00, Sat 09:00, Sun *09:00,
GMT

ASIA Mon-Fri, 08:00 to 17:00, Mon-Fri 12:00, Sat 09:00, Sun *09:00,
Malaysia Time Zone Malaysia Time Zone

*RMA’s deemed necessary by Technical Support will ship at the request of the customer on Sunday
prior to the cut off times. Two service types will be considered for shipment. Next Flight Out (NFO)
Service is subject to commercial airlift schedules and restrictions. Certain origins & destinations may
not have NFO service available for shipment. Orders unable to ship on Sunday will be processed
the following business day. If the customer is located within 100 miles in North America or 50KM
internationally, local courier services will be considered. Customer must be on site to accept delivery
2. Select the events you want to log.
for Sunday deliveries.
When you select an event level, all levels above the selection are
Additional RMA Information can be found at http://www.bluecoat.com/ included. For example, if you select Verbose, all event levels are
support/supportpolicies/rmainformation#process included.
3. Click Apply.
ProxySG Monitoring Eventlog Levels
There are several methods that can be used to monitor the ProxySG
appliance.
1. Event logging
2. SNMP
3. Health monitoring
For more information review the following sections in the “Configuration
Management Guide” Setting Event Log Size
1. SGOS 5.x: “Configuration Management Guide” You can limit the size of the appliances event log and specify what the
Volume 9: Managing the Blue Coat ProxySG Appliance, appliance should do if the log size limit is reached.
SUPPORT PARTNER Chapter 2: Monitoring the ProxySG
1. Select Maintenance > Event Logging > Size.
TROUBLESHOOTING GUIDE 2. SGOS 6.x: “SGOS Adminstration Guide”
Chapter 72: Monitoring the ProxySG
23
 USER GUIDE
2. In the Event log size field, enter the maximum size of the event log in
megabytes.
3. Select either Overwrite earlier events or Stop logging new
events to specify the desired behavior when the event log reaches
maximum size.
2. Click New to add a new e-mail address; click OK in the Add list item
4. Click Apply. dialog that appears.
3. In the SMTP gateway name field, enter the host name of your mail
Email Alerts
server; or in the SMTP gateway IP field, enter the IP address of
The ProxySG can send event notifications to Internet email addresses
your mail server. The ProxySG is configured to use only one of these
using SMTP. This setting applies to all events that can be configured to
two fields.
send mail such as health check warnings, CPU, memory pressure, disk
errors, etc. 4. (Optional) If you want to clear one of the above settings, select the
radio button of the setting you want to clear. You can clear only one
Note: The ProxySG must know the host name or IP address of your
setting at a time.
SMTP mail gateway to mail event messages to the e-mail address(es)
you have entered. If you do not have access to an SMTP gateway, you 5. (Optional) You can specify a custom address for email notifications
can use the Blue Coat default SMTP gateway to send event messages in the Custom `From’ address field. For example, headoffice.sg1@
directly to Blue Coat. bluecoat.com.
If set, all email notifications use the specified address (headoffice.
The Blue Coat SMTP gateway only sends mail to Blue Coat. It will not
[email protected]) as the sender’s address.
forward mail to other domains.
By default, the field is empty and email notifications use the Appliance
Enabling Event Notifications: Name configured on the ProxySG as the sender’s address. For
1. Select Maintenance > Event Logging > Mail. information on configuring the appliance name, refer to Volume 1:
Getting Started.
SUPPORT PARTNER
TROUBLESHOOTING GUIDE 6. Click Apply.

24
 USER GUIDE
Syslog • OK - The monitored system or device is behaving within normal
You must have a syslog daemon operating in your network to use syslog operating parameters.
monitoring. • WARNING - The monitored system or device is outside typical
Syslog format: Date Time Hostname Event. operating parameters and may require attention.

Many customers using syslog have multiple devices sending messages • CRITICAL - The monitored system or device is failing, or is far outside
to a single syslog daemon. This allows viewing a single chronological normal parameters, and requires immediate attention.
event log of all of the devices assigned to the syslog daemon. An event This value can be found at the top Right-hand side of the Management GUI
on one network device might trigger an event on other network devices,
which, on occasion, can point out faulty equipment.
Enabling Syslog Monitoring
1. Select Maintenance > Event Logging > Syslog.
2. In the Loghost field, enter the domain name or IP address of your log
host server.
3. Select Enable Syslog.
4. Click Apply. A change in health status does not always indicate a problem that
requires corrective action; it indicates that a monitored metric has
deviated from the normal operating parameters. The health monitor aids in
focusing attention to the possible cause(s) for the change in health status.
The ProxySG monitors the status of the following metrics:
• Hardware - Disk, Voltage, Temperature, Fan speed, Power supply
• System Resources - CPU, Memory, and Network usage
• ADN Status
• License Expiration and Utilization
• Health Check Status - health status of external services used by the
appliance

Health Monitoring Changing Threshold and Notification Properties:

Health Monitoring allows you to set notification thresholds on various 1. Select Maintenance > Health Monitoring.
internal metrics that track the health of a monitored system or
2. Select the tab for the metric you wish to modify.
device. Each metric has a value and a state. The value is obtained by
a. To change the system resource metrics, select General.
periodically measuring the monitored system or device. In some cases,
SUPPORT PARTNER the value is a percentage or a temperature measurement; in other b. To change the hardware, ADN status and health check status
TROUBLESHOOTING GUIDE cases, it is a status like “Disk Present” or “Awaiting Approval”. The state metrics, select Status.
indicates the condition of the monitored system or device: c. To change the licensing metrics, select Licensing.

25
 USER GUIDE
3. Click Edit to modify the threshold and notification settings. The Edit Information can also be found in the following sections of the
Health Monitor Setting dialog displays. Hardware, health check, and “Configuration Management Guide”
ADN thresholds cannot be modified.
1. SGOS 5.x: “Configuration Management Guide”
Volume 9: Managing the Blue Coat ProxySG Appliance,
Chapter 2: Monitoring the ProxySG
2. SGOS 6.x: “SGOS Adminstration Guide”
Chapter 72: Monitoring the ProxySG
Section DL Monitoring Network Devices (SNMP)
Obtaining MIB Files
The ProxySG uses both public MIBs and Blue Coat proprietary MIBs.
You can download the MIB files from the Blue Coat website.
To download the MIBs:
1. Go to https://bto.bluecoat.com/download
2. Select the “ProxySG” Product, then the Model Number of the device
you are configuring

Modify the notification settings.

• Log adds an entry to the Event log (and/or SYSLOG if configured).
• Trap sends an SNMP trap to all configured management stations.
• Email sends an email to the addresses listed in the Event log
properties.

SNMP
The ProxySG provides the capability to configure SNMP for single
network management systems (NMS), a multiple user NMS, and for
notification only.
Configuring SNMP
Full configuration information is found in the Configuration Management
Guide and the Help files on in the SNMP section on the SG itself.
SUPPORT PARTNER
TROUBLESHOOTING GUIDE 1. Maintenance > SNMP
3. Select the exact Model of this Hardware Version you are configuring,
2. Click “Help” on each tab for information on configuring SNMP then select the SGOS version that in installed on the device.
26
 USER GUIDE
4. The MIBS should be listed in the “Files” tab of the SGOS version • Sophos
chosen, such as below
• Panda
• Symantec*
• Trend Micro*
*Supported however no longer available on Blue Coat Price List for purchase

Note: To load the Blue Coat MIBs on an SNMP network manager, be

sure to load the dependent MIBs, as well. Most commercial SNMP-
based products load these MIBs when the software starts.

Proxy AV Basics
How the Proxy AV Works
The Proxy AV device is an explicit device which is either directly
connected to a proxy SG or connected via L2 switch on the network.
It functions as an Antivirus/Malware scanner for HTTP traffic which is
intercepted and forwarded to the AV device by a ProxySG or other 3rd
party device. Identifying and blocking Viruses, Worms, Trojans, Adware
and Spyware. Content Scanning Modes: Response modification and Request
modification.
The communication between the ProxySG/3rd Party device and Proxy
AV is performed via the ICAP protocol (1344 or 11344 for Secure ICAP). Most web malware deployments involve the use of a response
modification service (RESPMOD). A response modification service
Scanning outbound traffic (to the internet) or inbound traffic (from
analyzes inbound client requests. That is, the response that is fetched
the internet) before being cached on the ProxySG/Gateway device,
from the OCS is scanned for malicious content before it is delivered
to ensure that prevents Antivirus software/Malware from entering the
to the user who requested the content. If the content is verified as
customer’s network.
clean (and also allowable by corporate policy), the client receives
The supported Antivirus/Malware Scanning Engines the web objects (that comprise web pages). If malware scanning
SUPPORT PARTNER • Kaspersky detects malicious content, the response is quarantined, the objects
TROUBLESHOOTING GUIDE are not cached, the event is logged, and the client receives a message
• McAfee indicating that a virus was found.

27
 USER GUIDE
A request modification service (REQMOD) is typically used to scan This is the easiest to configure and deploy, however doesn’t offer any
outbound Web requests or Web mail attachments before users post redundancy if the Proxy AV should fail.
them to file servers such as Gmail and HotMail servers. A request
Multiple SGs to Multiple AVs
modification service mainly prevents data leak in an enterprise.
Proxy AV Deployment Types
It is recommended that the Proxy AV and Proxy appliances reside on the
same subnet, even when multiple ProxySG devices are load balanced
with multiple Proxy AV appliances.
The closer the two devices are the better the performance, the fewer
hops between the two the quicker the connection will be.
Direct Access
In most deployments the ProxySG and Proxy AV will have direct
Internet access, in these instances the AV device will be able to initiate
direct pattern file updates of the scan engine installed on the device
(The destination of the request/download will vary depending on the
vendor used on the AV device, Sophos/McAfee/Kaspersky etc etc), and
licensing updates from Blue Coat.
Closed Network
In some instances the Proxy AV will not be allowed to access the
internet directly (particularly in government or military environments),
which means that pattern file updates must be requested by a device ProxySGs configured with multiple AV devices via the use of Service
which is allowed internet access, and the Proxy AV must be configured Groups to load balance traffic across Proxy AV devices.
to fetch the update file from this device internally (device must be (For more information see the following KB article https://kb.bluecoat.
configured as a Web server for the AV to connect and fetch the update com/index?page=content&id=KB1470)
file via HTTP). Licensing updates from Blue Coat can be installed
manually via raw text on the management GUI of the device. General Data Gathering Guidelines
One SG to One AV Restart Issues
The Proxy AV (as with all hardware appliances) can suffer from reboot
issues, which can either be caused by physical hardware issue or
software issues with the code running on the device.
There is no core output (as with proxy SG) to refer to, so with the Proxy
AV a complete logs.zip file will be required to troubleshoot the issue.
AFTER the restart has occurred (ideally generate the logs.zip file within 1
hour of the restart occurring to capture the freshest data.)
SUPPORT PARTNER
TROUBLESHOOTING GUIDE To troubleshoot this issue the following debug data is required.
1. Logs.zip file from the Proxy AV device
28
 USER GUIDE
Collecting Data for Restart issues 2. Start PCAP on the AV
1. After restart has occurred Upload the logs.zip from the Proxy AV 3. Attempt to establish a connection between the two (by performing a
device (ideally within 1 hour of the crash) health check on the SG or by sending some network traffic through
the SG which matches AV scanning rules in policy), in order to show
False Positive
that connectivity is failing or is flapping.
The antivirus/malware engine installed on the Proxy AV device can
on occasion provide a False Positive result for a file that is scanned 4. Stop PCAP on the SG
(meaning that it identifies a Virus/Trojan/Worm/Spyware/Malware being
5. Stop PCAP on the AV
embedded or part of that particular file) when in fact the file is known to
be clean. 6. Upload PCAP and Sysinfo via the MC of the ProxySG

To troubleshoot this issue the following debug data is required. 7. Upload the PCAP and logs.zip from the Proxy AV

1. Packet Capture on the ProxySG device Failed Antivirus Updates

The antivirus/malware engine installed on the Proxy AV device
2. Logs.zip file from the Proxy AV device
periodically checks for updates to the engine, to keep it as up-to-date
Collecting Data for False Positive issues as possible, to protect against the latest threats.
1. Start PCAP on the SG On occasion these updates can fail to download/install. When this
occurs an error will show on the Management GUI page “Anti-Virus
2. Reproduce the issue
Settings”, see below for an example.
3. Stop the PCAP on the SG
4. Upload the PCAP from the SG via the MC
5. Upload the logs.zip from the Proxy AV device
Connectivity Issues
On occasion the ICAP connectivity between the SG and AV can flap or
fail completely. To troubleshoot this issue debug data will be required
from both the ProxySG and the Proxy AV, either during a period of time
when connectivity has dropped, or when the connectivity flaps.
To troubleshoot this issue the following debug data is required.
1. Packet Capture on the ProxySG device
2. Packet Capture on the Proxy AV device
3. Sysinfo from the ProxySG device (taken after pcaps taken and issue
seen) To troubleshoot this issue the following debug data is required.

4. Logs.zip from the Proxy AV device (generated after pcaps taken and 1. Packet Capture on the Proxy AV device when replicating issue
SUPPORT PARTNER issue seen)
2. Logs.zip file after replicating the issue
TROUBLESHOOTING GUIDE Collecting Data for False Positive issues
3. Packet Capture from ProxySG device (*if Proxy AV is configured to
1. Start PCAP on the SG access internet through a ProxySG device)
29
 USER GUIDE
Collecting Data for Failed Antivirus Updates. The number of Antivirus Engine processes running on the device can be
view via the following advanced URL on the Proxy AV Management GUI.
1. Start PCAP on Proxy AV
http://x.x.x.x:8082/enabled_features.html
2. Start PCAP on ProxySG *if AV is configured to access internet through
the ProxySG device Under NASA settings, see below.
3. Force an update of the Antivirus engine.
a. Browse to the Management GUI of the Proxy AV
b. Browse to “Antivirus” tab
c. Tick “Force Update” option and click “Update” button next to
installed Scan Engine Vendor in the list. (Similar to that below)
4. Stop PCAP on the Proxy AV
5. Stop PCAP on the ProxySG
*if AV is configured to access internet through the ProxySG device
6. Upload Proxy AV PCAP
7. Upload ProxySG PCAP from Management Console
8. Generate logs.zip on the Proxy AV and upload
Memory/CPU issues
High Memory Pressure or CPU spikes can occur on the Proxy AV device.
CPU issues can arise when the device is being over-utilised. This can
occur for many different reasons, such as scanning infinite streams,
or scanning large files, or in some rare instances it can occur when an
update of the Antivirus Engine is occurring.
To stop infinite streams and/or large files being sent to the Proxy
AV to be scanned please ensure that the ProxySG device has Best
Practices Policy installed on them. This is located in the “ProxySG/
ProxyAV Integration Guide (SGOS 5.5 and later and AVOS 3.2 and later)”
document which can be found on BTO here. To troubleshoot this issue the following debug data is required.

For Memory Pressure issues generally these are caused by the AV 1. Logs.zip (generated within 1 hour of the issue occurring –or– during
scanning engines running in memory at the time. the spike occurring if possible)

Check the number of Engine processes (actual number of physical Collecting Data for High Memory Pressure/High CPU issues
Antivirus engines running) configured to run in memory on the AV device. 1. After/During CPU/Memory Pressure spike has occurred Upload the
SUPPORT PARTNER It should be set to the Default value, but sometimes this is altered. If it logs.zip from the Proxy AV device (ideally within 1 hour of the spike or
TROUBLESHOOTING GUIDE has been changed to values higher than the Default, set back to Default during the spike).
and monitor the device.
30
 USER GUIDE
Proxy AV Monitoring Email
There are several methods that can be used to monitor the ProxyAV To Configure Email alert notifications select “Alerts Settings” from the
appliance. “Alerts” screen (see above)

1. Email
2. Logging
3. Syslog
4. SNMP Trap
For more information review Chapter 4: Configuring Antivirus Scanning,
Section E: Enabling and Configuring Alerts in the “Configuration
Management Guide” for the Proxy AV Appliance available on BTO.
To enable Alerts and SNMP Traps in the Management Console of the
Proxy AV, select “Alerts”

Fill in the relevant fields.

Sender email address – The Address that identifies to the reader which
appliance is sending the notification
Recipient email address – The Address of the recipients of the messages
SMTP server address – Enter the Server IP address or name (example
mail.example.com)
If authentication is required enter the relevant details.
Syslog
Syslog server details can be configured on the same page as above
(“Alerts” menu then “Alert Settings”)

SUPPORT PARTNER
TROUBLESHOOTING GUIDE
Select the Alert or SNMP Trap for each event as required.
31
 USER GUIDE
Enter the server IP address, port that the server is listening on and the From here the SNMP service can be enabled and disabled, SysLocation
TLS settings used (if necessary) and SysContact details are configured. The Trap community is defined,
along with the interface the traps will be sent from and the IP address of
SNMP Traps
the SNMP Manager device it will send the traps to.
For the Proxy AV device to send SNMP Traps when required, the device
SNMP Version and Read Community string is also configurable as
needs to have an SNMP manager configured.
necessary.
This can be done via the “Advanced” menu, then “SNMP” Settings. See
Logging
below
This creates an entry in the AlertLogFiles.log file for each alert
configured on the “Alerts” page.
The AlertLogFile.log can be located on the “Log Files” Page.

Director
How Director Works
The Blue Coat Director device gives Administrators the power to control
and monitor all SG Appliances installed on their network from one
single device. Via the Web GUI interface of Director you can configure,
monitor and enforce security and acceleration policies on multi devices
simultaneously.
Its reporting functionality allows you to keep track of key health
statistics, performance per region and job status.
It allows you to gain remote access to SG devices connected from one
central location, to make individual changes. Alternatively you can make
changes to multiple SG devices at once using “Overlays” or “Profiles”
which are pushed to all desired devices. Using port 22 (SSHv2 Simple)
to connect directly to SG devices.
Authentication to the Management GUI of Director unit is performed
over port 8082, however to console is controlled by a Java Applet which
launces on your PC and connects to Director on port 8085.
The Java GUI output can be seen below; it has 4 main tabs which are
used to perform all the different tasks and functions the device can
perform.

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

32
 USER GUIDE
Please ensure ICMP ping connectivity is not blocked and possible
between the Director and SG, as Director uses ICMP to connect to new
SG devices.

For more information on how to configure the Director unit please The following information is required for Director Connectivity Issues
see the Configuration and Management Guide for the SGME version 1. Packet Capture on Director
required available here (this will require a BTO login)
2. Packet Capture on SG (if connectivity issue is between SG and
Gathering Troubleshooting Data. Director)
The same 2 pieces of debug data are required to troubleshoot any/all 3. Messages file (Local Logging Level set to Debug)
Director related issues. Those are listed below
4. Debug Dump
1. Messages file (Event Logging)
Collecting Data for Director Connectivity Issues
2. Debug Dump
1. Start PCAP on Director
Information on how to generate these can be found in the Appendix.
2. Start PCAP on SG (if required)
Other issues, such as those below, require more diagnostic information.
3. Reproduce the issue
Please see these examples for confirmation of what is required.
4. Stop PCAP on Director and upload
Connectivity Issues
Please ensure firstly that none of the following ports are blocked by any 5. Stop PCAP on SG (if required) and upload via the MC
FW or IDS device on the network, as Director is reliant on these ports for
6. Generate Debug Dump on Director Upload
general access and running functionality of the device.
SUPPORT PARTNER
TROUBLESHOOTING GUIDE

33
 USER GUIDE
Overlay or Profile issues Audit Logging
The Following Information is required for Overlay or Profile issues Audit Logging tracks the contents of Profiles, Overlays, Configuration
1. Packet Capture on Director and content jobs, backups when executed. As well as the Username
and IP address of the user who executed them.
2. Packet Capture on SG
This file is generally the one requested by Blue Coat support rather than
3. Messages file (Local Logging Level set to Debug) the Event Logging file, as it contains more data.
4. Audit Logging File
SNMP Monitoring
5. Copy of Profile or Overlay
SNMP configuration of the Director unit is only available via the CLI.
6. Debug Dump
From here you can configure the following
7. Screenshot of Error message seen in Director console (if any)
director (config) # snmp-server ?
Collecting Data for Director Connectivity Issues
community: Set read-only community name
1. Set Local Logging level to “Debug” contact: Set contact string
enable: Enable certain SNMP server options
2. Ensure the
host: Specify hosts to receive SNMP notifications
3. Start PCAP on Director inform: Set certain SNMP inform options
4. Start PCAP on SG (Filtered to Director IP address) location: Set location string
traps: Set certain SNMP trap options
5. Push Overlay or Profile to replicate the issue
The Director can send the following SNMP traps to the configured
6. Take a screenshot of the error given SNMP server
7. Stop PCAP on Director and upload
8. Stop PCAP on SG and upload
9. Upload Messages file from Director
10. Set Local Logging level to “Notice”
11. Generate Debug Dump on Director and upload

Director Monitoring
Logging
Director has two types of logging which help you monitor the use of the
device for troubleshooting purposes.
Audit Logging (Messages File)
SUPPORT PARTNER Event Logging
TROUBLESHOOTING GUIDE Both are configured to be enabled by default.

34
 USER GUIDE
Reporter Other issues, such as those below, require more diagnostic information.
Please see these examples for confirmation of what is required.
How Reporter Works
Blue Coat Reporter analyses log files from one or more ProxySG Memory Related Issues
appliances, generating rich reports that can categorize content, capture Most commonly occurs on Windows 32-bit server, but can occur on
user browsing activities on the Internet, and report on traffic volumes all platforms. If this does occur on a 32-bit version of OS it’s likely that
and flows. the solution would be to upgrade the OS to a 64-bit version in order to
By processing SG access logs into its proprietary database reports can address more physical memory.
be generated per device or collectively via its Management Console. Errors like the following are seen in the Reporter Journal File
Running on Windows 2008 or Red Hat Enterprise Linux 5 platforms
there is both a 32-bit and 64-bit version (including Windows 2003 64-bit
ONLY); couple with a recommended sizing guide in order give a stable
platform to work from. Please see below

The following information is required for Reporter Memory Related

Issues
1. Reporter Journal File
2. Reporter Diagnostic File
Collecting Data for Reporter Memory Related Issues
Collect and upload the debug data AFTER the Memory issues are
reporter (even whilst they are still occurring).
For information on how to collect the Reporter Journal File please see
Reporter License comes in three separate forms. Appendix N: Reporter Journal File

Standard Edition – Supports up to 50 Million Log Lines – Free of Reporter Crash issues
Charge to SG customers with valid support The following information is required for Reporter Crash Issues
Enterprise Edition – Supports up to 2.5 Billion Log Lines – Chargeable For Windows
Premium Edition – Supports up to 20 Billion Log Lines - Chargeable Run Dr. Watson debugger for Windows 2003 server version. Create a
user dump file for further analysis.
Gathering Troubleshooting Data.
For ALL Reporter issues the same piece of debug data will be required Collect application memory dump for Windows server versions 2008,
to help diagnose the issue, and should be generated AFTER the issue Vista or Windows 7 systems
has occurred (whatever that may be). The piece of debug data is called For Linux
the Reporter Diagnostic File.
SUPPORT PARTNER Start Reporter and try and reproduce the crash symptoms
To generate this File please see Appendix M: Reporter Diagnostic File.
TROUBLESHOOTING GUIDE Collect memory dump

35
 USER GUIDE
How To Create Crash Dumps? Linux
Windows 2003 1. Stop reporter.
1. Stop reporter. a. To do this you execute a “./bcreporter stop” in the etc/init.d folder.
a. To do this you execute a “./bcreporter stop” in the etc/init.d folder. b. To make sure it’s stopped you execute a “./bcreporter status” in the
same folder.
b. To make sure it’s stoped you execute a “./bcreporter status” in the
same folder. 2. Run reporter in a command window.
2. Run reporter in a command window. a. Open a command line window and navigate to the opt/bc/reporter
folder.
a. Open a command line window and navigate to the opt/bc/reporter
folder, or your installed location. b. Execute the “ls -l “ command to procure a directory listing. Along
with other folders and files, you should see the bcreporter binary, a
b. Execute the “ls -l “ command to procure a directory listing. Along
few files ending with the .out extension, and a isready.txt file.
with other folders and files, you should see the bcreporter binary, a
few journal_xxxx.tst file or two , and a isready.txt file. c. Run these commands
c. Run these commands i. ulimit -c unlimited
i. ulimit -c unlimited ii. ./bcreporter
ii. ./bcreporter d. Reporter will now be running, and functional, and you should be
seeing it log events to the same terminal window.
d. Reporter will now be running, and functional, and you should be
seeing it log events to the same terminal window. e. If it goes into a high utilization situation again, you need to execute
this command in another terminal window, to determine the PID
e. If it crashes again, the dump file, called “core”, will be in this folder.
number:
Please upload to this to your SR for analysis by the Blue Coat team
a. –” ps –aux |grep reporter”
Windows 2008
2. 6: In the same terminal window, use the number found in the
Dump settings for system failures are located in the System Properties above command to execute this command which will kill the
| Advanced tab | Startup and Recovery Settings. You no longer run the Reporter service, and create a core:
drwatson32 exe to configure it. a. –” kill -segv <PID NUMBER> “
There is more information from Microsoft at these two links: 3. General LINUX Notes on taking a core:
http://msdn.microsoft.com/en-us/library/bb787181%28VS.85%29.aspx 4. To setup your LINUX server to permanently provide Memory dumps
- Collecting User-Mode Dumps (Notice that user-mode and system- for all applications, follow these two steps.
service dump files are NOT located in the same place.)
a. 1: Add below 2 lines in “/etc/security/limits.conf”
http://support.microsoft.com/kb/931673 - How to create a user-mode b. * hard core unlimited
process dump file in Windows Vista and in Windows 7 * soft core unlimited
http://support.microsoft.com/kb/949180 - How to create a user-mode c. 2: comment out the below line in “/etc/profile” if it is not
process dump file in Windows Server 2008 commented by symbol “#”
SUPPORT PARTNER d. TIP: In other words, change
http://support.microsoft.com/kb/286350 - How to use ADPlus to
TROUBLESHOOTING GUIDE e. ulimit -S -c 0 > /dev/null 2>&1
troubleshoot “hangs” and “crashes”

36
 USER GUIDE
f. to
g. #ulimit -S -c 0 > /dev/null 2>&1
5. The “/proc/sys/kernel/core_pattern” file contains a pattern for the
name of the core dump file and defaults to the value “core”.
6. The “/proc/sys/kernel/core_uses_pid” file contains a flag designating
whether or not to append a dot followed by the PID of the dumping
process.
7. The core_pattern may contain certain % specifications that cause
other elements to be included in the name of the core dump file.
8. Here are the possible % specifications:
9. %% -- output one ‘%’
10. %p -- pid
11. %u -- uid
12. %g -- gid
13. %s -- signal number
14. %t -- UNIX time of dump
15. %h -- hostname
16. %e -- executable filename

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

37
 USER GUIDE
APPENDIX A: OPENING SERVICE REQUESTS
BlueTouch Online **SRs can still be raised using a Serial Number which has expired
support, utilizing a one-time exception. However devices are only
Creating a New SR on BTO
eligible for this exception if the Support Contract has expired within the
It is important to provide a clear and full overview of the issue and
previous 1 month.
provide the necessary files (per type of issue) when logging a SR on
Blue Touch Online. If it has expired outside this time you will receive the following error
when attempting to create the SR using the Serial Number.
Create a New SR
1. “Verify Serial Number” Page
Enter the Serial Number of the Device by using the “Lookup” Button
highlighted.
Click “Next Step” Button
2. “Summarize Issue” Page
Once selected please enter the following fields to accurately portray the
type of issue being experienced along with a description of the severity.
This will ensure that the SR is worked upon by the engineer with the
best skills and knowledge to resolve the problem as quickly as possible.

Enter the Serial Number and click “Search” once returned click “Select”
to use this device to open a new SR.

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

38
 USER GUIDE
The “Subject” field should be no longer than 100 characters and should Please fill in ALL questions with as much detail as necessary, as this
describe the exact problem being experienced. information will be vital for the Blue Coat Engineer who works on the SR.
Alternatively this information may be requested later which may impact
The problem “Description” field should follow the below format, with
resolution times.
a description of the issue in each section. Do not forget to include the
troubleshooting steps taken prior to raising the SR with Blue Coat and Click “Next Step” Button
make sure the following points are included.
5. “Attach Files” Page
• When did the issue start?
Upload ALL relevant Debug Data regarding the issue into the SR
• How often does it occur?
• Is this a new or existing setup and was it ever working?
• How “exactly” does the end user experience the problem?
›› Provide URLs and screen shots of errors where possible
• What applications are involved?
• What devices are involved?
• Precise, detailed, step-by-step, duplication steps.
• What is the impact to production?
• What debug data has been created?
• Notes on the troubleshooting performed so far and what has been
learned regarding
Click “Next Step” Button
The “Problem Area” field which is located in the “Summarize Issue”
3. “View Solution” Page page (page 2), contains high level categories which help determine what
A list of relevant solution documents will be listed; these are returned kinds of questions asked/files to be gathered. These are defined on the
based on Key Words defined in the “Summary” field entered on the right hand side of this “Attach Files” page).
“Summarize Issue” page, and the Product Type (SG, AV, VA, PS etc) You will need to Upload or Skip each “Requested File” before clicking
If none are relevant, or no results are returned skip to Next Step.

Click “Next Step” Button Click “Next Step” Button

4. “Refine Issue” Page

Depending on the problem “Type” selected from the “Summarize
Issue” page, there will be a list of predefined questions generated, in
order to give more detailed information regarding the background of
SUPPORT PARTNER
the issue.
TROUBLESHOOTING GUIDE

39
 USER GUIDE
6. “Add Preference” Page Blue Coat introduced a new Phone System in 2013, from which you can
be routed to your SRs owner directly, simply by entering the SR number
via your phone during the menu options.
You can obtain the current Status of the ticket and schedule a call back
from that engineer at any specified time if they are unavailable. You have
the option to speak to another available engineer if required.

All numbers are listed on the Main Blue Coat website here
http://bluecoat.com/company/contact/contact-service-support

Americas: Europe, Middle East and

Technical Support: Africa:
+1 408-220-2270 Technical Support:
+1 866-362-2628 (toll-free) +44 (0)1252 554 700
Duty Manager: Duty Manager (UK):
+1 408-541-3700 44 1252 554710
+1 888-216-6833 (toll-free) 0800-030-4183 (toll-free)
Customer Care: Customer Care:
Enter in the Preferred Collaboration details, ensuring to enter any direct +1 408-220-2270 +1 408-220-2270
Telephone numbers or alternative email addresses you would like to use +1 866-362-2628 (toll-free) +1 866-362-2628 (toll-free)
for communication regarding this new SR.
Click “Next Step” Button Latin America: Asia-Pacific:
Technical Support: Technical Support:
7. “Confirm Submission” Page
+1 408-220-2270 +60 3 2687 7501
Review all data entered, if correct please click 1-800-881-594 (toll-free)
Duty Manager:
Click “Next Step” Button +1 408-541-3700 Duty Manager:
8. “Complete” Page +1 408-541-3700
Customer Care:
1-800-881-595 (toll-free)
The SR will be created shortly and an email confirmation sent. +1 408-220-2270
Customer Care:
The SR can be viewed, altered and monitored via the main “Service Mexico: +1 408-220-2270
Management” page on bto.bluecoat.com. Here Technical Support: +1 866-362-2628 (toll-free)
+1 408-220-2270
Telephone + 0 1800 083 4989 (toll-free)

For network down or other emergencies, please contact your Blue Duty Manager:
SUPPORT PARTNER Touch Support Partner or call our Global Support Centers directly at the +1 408-541-3700
TROUBLESHOOTING GUIDE numbers below. The online portal (Blue Touch Online) is intended for P3 Customer Care:
and P4 issues only. +1 408-220-2270
40
 USER GUIDE
Updating a Service Request
It is possible to update a SR either through the web portal on BlueTouch
Online (BTO) or through email.
To update an existing service request through email then it is important
to use the following syntax:
• [email protected] Destination Email Address
• {SR#:2-10101010} Subject line must contain the SR number in the
specified format.
This will update the service request with the new information and send
an email to the engineer working on the case.
If you have had direct contact from the engineer responsible for your
case you should ensure to include his email address on this email also.
To ensure he sees the update.
There is not an option to open a new service request through email.

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

41
 USER GUIDE
APPENDIX B: MANUALLY RETRIEVING FILES FROM THE PROXYSG
If you are unable to upload service information using the Management Example output
Console (GUI), the following steps will allow you to pull the information
locally, for uploading through an alternate channel.
1. SysInfo
a. In the browser enter: https://x.x.x.x:8082/sysinfo
b. Use your browser’s FILE | SAVE and save as TEXT
2. Event Log
a. In the browser enter: https://x.x.x.x:8082/Eventlog/
fetch=0xFFFFFFFF
b. Use your browser’s FILE | SAVE and save as TEXT
3. Packet Capture For more information see Appendix F: Cores
a. In the browser enter: https://x.x.x.x:8082/PCAP/statistics
b. Click the download link
c. Save the .cap file locally
4. Snapshots
a. In the browser enter: https://x.x.x.x:8082/Diagnostics/Snapshot/
b. Click “download all” beside each snapshot
c. Save the .gz file locally
5. Context/Core Image
a. In the browser enter: https://x.x.x.x:8082/CM/Core_image
b. Click the hyperlinks to download the core files.

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

42
 USER GUIDE
APPENDIX C: UPLOADING FILES TO FROM PROXYSG BLUE COAT
There are different Methods for uploading files to technical support upload.bluecoat.com
listed here in order of preference.
1. Enter your name, email, SR number, and browse to the file
Those include...
2. Click “Upload File to Support”
The file will be uploaded to a folder with your SR number as the name
ProxySG Management Console GUI (MC) which is accessible to Blue Coat engineers.
1. Maintenance > Service Information > Send information > Send service
information
2. Enter the service request number (include the dash (-))
3. Press the button “Select Newest” –OR- check individual boxes for
desired files
4. See how to do this using the CLI below.

ftp.bluecoat.com
1. Use only if the MC or https://upload.bluecoat.com are not available
2. Use Anonymous credentials
a. Change the directory to \incoming\support_dir (note that directory
reading is not allowed)
b. Make a new directory (mkdir) named the same as SR# (or use
another pertinent name)
c. Change the directory to the one that was created in the previous
step
d. “Put” the file into this directory (be sure the FTP client is in binary
transfer mode)
e. Forward the name of the new directory to the Blue Coat technical
support engineer
SUPPORT PARTNER
TROUBLESHOOTING GUIDE

43
 USER GUIDE
Uploading Files Using the CLI
Commands as follows
>en
Enable Password:
#config t
#(config)diagnostics
#(config diagnostics)service-info
#(config service-info)view available
Name Approx Size (bytes) URL
Event_log 1,245,184 /Eventlog/fetch=0xFF0000FF
Policy_trace 4,332 /Policy/Trace/default_trace.html
System_information Unknown /SYSINFO/text
Snapshot_sysinfo 66,127,903 /Diagnostics/Snapshot/sysinfo/download/all
Snapshot_sysinfo_stats 115,586,669 /Diagnostics/Snapshot/sysinfo_stats/download/all
Access_log_main 842,828 /Accesslog/dump_log/main
Access_log_ssl 617,112 /Accesslog/dump_log/ssl
Access_log_test 327 /Accesslog/dump_log/test
#(config service-info)send 2-123456789

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

44
 USER GUIDE
APPENDIX D: HOW TO CREATE A POLICY TRACE ON THE PROXYSG
A policy trace will track and log what policy rules are executed and Tracing requests from one Client IP address
which are not for any given request. This can be very helpful for tracking
1. Go To: https://x.x.x.x:8082/policy
down URL, authentication, or any other issue that might be policy
a. Click: Delete all policy traces
related. The SG has a default policy trace setting which captures all
policy execution from every device passing through it, but it is also 2. Open VPM (Visual Policy Manager)
possible to limit policy tracing to just a few devices or even just one a. Configuration > Policy > Visual Policy Manager > Launch
device. In most cases it is not helpful to trace policy for every box on
the network. When a policy trace is requested it is assumed that the 3. Add new Web Access policy tab
trace will be limited to just one device (via IP address). a. Policy > Add Web Access Layer
4. Set “SOURCE” to the test client IP address
Default Policy Trace a. Right click in the source field
Via the Management GUI Browse to b. SET
c. NEW
Configuration > Policy > Policy Options
d. Client IP address/subnet
Choose “Trace All Policy Execution”
i. Enter IP address of the workstation used to test
Apply ii. Enter full subnet mask (255.255.255.255)
e. Click ADD > CLOSE >OK

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

45
 USER GUIDE
5. Set “ACTION” 7. Install Policy
a. Right click in the action field
b. DELETE (this will set it to “none”)

6. Set “TRACK”
a. Right click in the TRACK field
b. SET
c. NEW Your trace should look similar to this when complete:
d. TRACE
i. Rule and request tracing
ii. Check “Trace File”
iii. Name it (“MUST NOT” contain a space or the policy will not be
traced)
e. Click OK > OK

Saving the policy trace

1. https://x.x.x.x:8082/policy
2. Trace will display with the name you gave it previously.
SUPPORT PARTNER 3. Click the trace to open it.
TROUBLESHOOTING GUIDE
Save using FILE | SAVE and save as TEXT.

46
 USER GUIDE
APPENDIX E: HOW TO CREATE A PACKET CAPTURE ON THE PROXYSG
Overview Methods
When taking a packet capture it’s important to remember that the SG is Management Console
in the middle of the connection. This is the preferred method of taking a packet capture as it allows
ProxySG filtering and setting capture preferences.
1. Maintenance > Service Information > Packet Captures
Client OCS
2. Leave the defaults (no filter) and click “Start”
3. Select an optimal configuration for the problem being captured (see
In order to see all the information both sides of the connection must
“Start capture window”)
be captured (client and server side). Configuring a filter for the client IP,
for example, would only capture half of the traffic we need to see and 4. Run a “simple” duplication of the problem.
therefore render the packet capture useless.
5. Click “Stop”
Capturing with a filter for both the client IP and server IP (or server
6. The packet capture can then be downloaded to the PC for upload
hostname) can be effective depending on the circumstance; however,
via https://upload.bluecoat.com or uploaded directly to the case via
in most cases it is optimal to take an “unfiltered” packet capture. This
use of the “packet capture” check box in “maintenance > service
helps ensure we don’t miss anything.
information > send information > send service information.
Limitations
Start Capture Window
The packet capture utility in the SG has a few limitations that are vital to
take into account when capturing data.
1. The SG has a 100MB buffer. By default, once the buffer is full it will
stop capturing data. In a busy environment the buffer may fill very
quickly (as little as 6 seconds in some cases). In these cases it is
necessary to either filter on specific traffic to capture or reduce the
amount of traffic being captured (off hours duplication, etc.). When
using filters we may miss the data we need to see. It is preferable to
keep duplications as simple as possible and use unfiltered packet
captures.
2. It is not possible to capture properly using a filter for WCCP GRE
traffic. When using WCCP redirection with GRE an unfiltered packet
capture is required.
3. The SG will not save more than one packet capture at a time. Once
a packet capture is taken it must be either uploaded to the case or
SUPPORT PARTNER downloaded to the PC (saved) before any further packet captures are
TROUBLESHOOTING GUIDE taken. Any additional packet captures will overwrite the capture before
it as will downloading any running packet capture.
47
 USER GUIDE
• Capture all matching packets Packet capture information:
›› Captures all packets matching the “capture filter”. Packets captured : 0
Bytes captured : 0
• Capture first……matching packets
Packets written : 0
›› Packet capture will stop after reaching n number of packets. Bytes written : 0
• Capture last……matching packets Coreimage ram used : 0 B
Packets filtered through : 0
›› Once stopped the packet capture will save the last n packets
captured. For example, if 100,000 is specified the SG will save the Start packet capture
“last” 100,000 packets captured.
Stop packet capture
• Capture first……matching KBytes
Download packet capture file
›› Same function as previously mentioned, but based on size rather
than number of packets. Filters
• Capture last……matching KBytes The PCAP utility on the SG uses TCP dump style filter syntax for
capturing data (as does Director and Wireshark).
›› Same function as previously mentioned, but based on size rather
than number of packets. PCAP FILTER EXPRESSIONS
FILTER EXPRESSION PACKETS CAPTURED
• Save first…..bytes of each packet
ip host 10.25.36.47 Captures packets from a specific host with IP address 10.25.36.47.
›› Truncates each frame by the number of bytes specified rather than
not ip host 10.25.36.47 Captures packets from all IP addresses except 10.25.36.47.
capturing the entire packet. This can be used to reduce capture size
ip host 10.25.36.47 and ip host Captures packets sent between two IP addresses: 10.25.36.47 and
while maximizing the sample of packets captured. Used to capture 10.25.36.48 10.25.36.48.
packet header information when the payload (data) of the packet
ether host 00:e0:81:01:f8:fc Packets sent from one of these addresses to other IP addresses
is inconsequential to the issue. More packets will be captured in a are not filtered. Captures packets to or from MAC address
packet trace. 00:e0:81:01:f8:fc:.

port 80 Captures packets to or from port 80.

• Include…..K Bytes in core image
ip sr www.bluecoat.com and Captures packets that have IP source of www.bluecoat.com and
›› Used to insert packet trace data into a full core. ether broadcast ethernet broadcast destination

host www.bluecoat.com Captures packets which are sent/destined for the IP addresses
Browser URL which resolve to www.bluecoat.com

• https://x.x.x.x:8082/PCAP/statistics
• Limitation: Filters cannot be used. Ability to stop, start, and download More filter information:
the packet capture only. 1. http://wiki.wireshark.org/CaptureFilters
Example view of the PCAP URL: 2. http://www.ethereal.com/docs/eug_html_chunked/
ChCaPCAPtureFilterSection.html
Packet Capture Statistics
Current state: Stopped 3. http://www.cs.ucr.edu/~marios/ethereal-tcpdump.pdf
SUPPORT PARTNER Filtering: Off
TROUBLESHOOTING GUIDE

48
 USER GUIDE
APPENDIX F: CORE FILES ON THE PROXYSG
Overview The default setting is for “Context Only” as this provides sufficient
information in many cases and is by far the fastest type of core to write
A core is the process that occurs when the system writes, or “dumps”
resulting in the least amount of down time. A context can be written with
its memory to the hard drive. It may be referred to as a dump, core,
almost no more down time than it takes to perform a normal reboot,
context, core dump, image, core image, full dump, full core, or memory
but a full memory core can take as much as 5-20 minutes to write…
core. Blue Coat will normally use the following terms…
depending on your hardware and amount of memory. During the time
1. Full Core (full memory core) the proxy is writing a full memory core the box is offline and unavailable
2. Context (partial memory core) until it finishes and restarts (reboot).

3. Minicontext (Short text output logged in the sysinfo at the time a

core is written containing running processes and restart codes as well Core Generation
as other functions helpful to support.) Cores are generated in one of two ways.
While cores are not necessary or even helpful in troubleshooting all 1. Automatically
issues, they are required in order for Blue Coat to resolve certain issues
2. Manually (also referred to as “forced”)
such as system crashes (also referred to as “restarts”). Cores can also
provide information and insight into issues such as proxy hangs, high When the proxy restarts (crashes) due to some error, by default it will
CPU, and high memory usage. write a context and minicontext file automatically. It is also possible to
manually force the proxy to write a core.
Basic Configuration Forcing a Core
1. Set the core type (example is for “full core”) Useful information can be obtained from a core when the proxy is hung,
a. Click Maintenance > Core Images slow, or behaving unexpectedly (i.e. high CPU or high Memory usage).
For such cases the admin can force a core to be written to disk using
b. Select “full”
one of the following 2 methods:
c. Select how many cores you wish to store (default is 2)
d. Click APPLY 1. Command Line Interface (CLI): restart abrupt

2. Set the proxy restart mode 2. Serial Session: ctrl+x ctrl+h

a. Click Maintenance > System and Disks > Tasks **DO THIS ONLY ONCE!! - it may seem like nothing is happening, but it
b. Click “Hardware & Software” is dumping the core. If ctrl-x ctrl-h is entered a second time it will
overwrite the first core and the dump will be useless.**
c. Click APPLY
Note: The SG will write a core regardless of the “restart mode”.
However, best practice is to have the SG do a “hardware & software” **IMPORTANT**
restart unless instructed otherwise. Forcing a core is only useful if it’s accomplished while the proxy is in
a problem state such as when it is “hung”, experiencing slowness,
SUPPORT PARTNER
or behaving unexpectedly. If the ProxySG is actually crashing, or
TROUBLESHOOTING GUIDE
automatically restarting then a forced core “after the fact” will not

49
 USER GUIDE
provide any useful information. Forcing a core on a system that is https://<x.x.x.x>:8082/CM/Core_image
currently running normally will not provide any useful information. If you
See the example output of this page below:
are unsure about when a core should be forced, please consult with
support first.
PREREQUISITE FOR BOTH AUTOMATIC AND MANUAL
GENERATION OF A FULL CORE FILE
Full Core option must be enabled BEFORE manually forcing the Core file
via either of the methods mentioned above.
This can be done via the GUI or CLI The most recent restart is listed at the top of the table.
1. CLI Information regarding the SGOS version running at the time of the
Via the following CLI commands restart, along with HW & SW Exception codes, Page Fault address,
Process group and Process that the restart occurred in.
#(config)restart core-image full
The hyperlinks to download the various crash outputs are located in the
2. GUI
second column from the right.
Browse to the Maintenance Tab; select Core Images from the menu on
There will always be a “Details”, “Minicontext” and “Context” output
the left, select “Full” option from the list in the main screen. Such as that
from every restart.
below.
However *IF* Full Core is enabled on the SG device the addition of the
“Full” hyperlink will also be visible (as seen in example above). “Full”
Core also contains a “Context” core within, so only “Full” is required
when available.
Clicking on the link will prompt the download of the file to your local
machine. Save this file then upload into the Blue Coat ticket via BTO or
the upload website.

Files to Upload
(Maintenance > Service Information > Send Information > Send service
information)

Core Location/Retrieval 1. Context Core present

a. Sysinfo, Event log, Snapshots (all)
Normally it is sufficient to simply go to “Maintenance > Service
Information > Send information > Send service information”, check the b. Context
appropriate boxes, and click “Send” to upload the core. However, there 2. Full Core present
are times when it may be necessary to download the files manually.
SUPPORT PARTNER a. Sysinfo, Event log, Snapshots (all)
When manual retrieval is necessary the following URL may be used to
TROUBLESHOOTING GUIDE b. Full Core
download the core files.
c. Packet Capture (if applicable)
50
 USER GUIDE
Full Core with Packet Capture Included If the system settings already match sections 1 & 2 and restart has
already occurred
1. Set core type
a. Click Maintenance > Core Images 1. Download core from: https://x.x.x.x:8082/CM/Core_image

b. Click the radio button to set the core image to “Full” 2. upload.bluecoat.com - include sysinfo, event log, snapshots (all)
c. Click APPLY to store these settings. Options for forcing a core image
2. Set the proxy restart mode 1. Command Line Interface (CLI): restart abrupt
a. Click Maintenance > System and Disks > Tasks
2. Serial Session: ctrl+x ctrl+h
b. Click “Hardware & Software”
c. Click “APPLY” to store these settings.
Definition of Terms
3. Setup a packet capture
FULL CORE Written only by the newer platforms such as the 210s, 510s, 810s, and
a. Click Maintenance > Service Information > Packet Captures 8100s. Contains the data structures that are missing in the context and
b. Click “Start capture” is made up of the context and memory core in one easy file. Requires
10-20 minutes to complete. Most comprehensive core available.
c. Check “include………..K Bytes in Core Image” and set this to 1024
MEMORY Written by older platforms such as the 200s, 400s, 800s, 8000s.
d. Check “capture last….matching packets” and set this to 200000 CORE Contains data that is missing in the context. A memory core must
e. Click “Start capture” be matched up with the right context (both would be needed). As
f. Let run until restart happens and full core is written mentioned before we only configure for “full” or “context”. In this case
the “full” core is made up of a “context” and “memory” core.
Note: A full core may take as much as 5-20 min, depending on hardware
CONTEXT A partial dump of the system memory. Contains the data structures
and the amount of memory. minus the data.
Alternatively use the following CLI commands MINICONTEXT Short text output logged in the sysinfo at the time a core is written
containing running processes and restart codes as well as other
ProxySG>en functions helpful to support. Normally you won’t be asked to provide
ProxySG#pcap start coreimage 10000 last count 100000 this as it is contained in the sysinfo, but there are occasions when you
may be requested to obtain this manually from https://x.x.x.x:8082/
CM/Core_Image
Quick Reference RESTART System crash.
Normally referred to as a restart because when the system experiences
1. Set the proxy restart mode
a critical error it will automatically write a core image and “restart”.
a. Maintenance > System and Disks > Tasks -> Maintenance Tasks
CLI Command Line Interface.
2. Set core type Accessed via SSH to <proxyIPaddr> and port 22. Often a program
a. Maintenance > Core Image such as “Putty” is used. Also accessible via a hyper-terminal or other
terminal emulation such as TeraTermPro which makes use of the
3. Allow system to automatically restart COM port for a serial connection to the SG. Simply connect the serial
cable, open the COM port, and press enter 3 times to activate the CLI
4. Download context and memory core from: https://x.x.x.x:8082/CM/ console.
SUPPORT PARTNER Core_image
TROUBLESHOOTING GUIDE 5. upload.bluecoat.com – include sysinfo, event log, snapshots (all)

51
 USER GUIDE
APPENDIX G: PROXYSG HOW TO…
Archive and Restore the ProxySG Configuration Restoring the Configuration
It is always a good idea to backup the current configuration before If this is a new system (no IP configuration) you must first make a
making changes, especially major changes, to the ProxySG. The connection to the SG via a serial cable and run through the initial setup
ProxySG configuration is stored in a single text file and can be backed wherein the IP information is configured. Then begin with step 1 below.
up and restored with ease. 1. Launch the GUI management console

Backing up the Configuration 2. Restore the “configuration-passwords-key”.

(see: “Export- Import SSL Keys”)
1. (optional) Request a copy of your license from Blue Coat in advance
(1-2 days) 3. Restore other SSL keys. (see: “Export- Import SSL Keys”)
a. You can license your box via the management console after the 4. Download the content filtering database.
restore is complete.
5. Restore the system configuration
i. Maintenance>License>Install
a. Configuration > General > Archive > Install Configuration from:
ii. Click “retrieve”
b. Select “Local File”
b. However, just in case this fails for any reason it would be a good
c. Click “Install”
idea to have your license on hand so that you can install using the
local file. d. Browse to where you saved the backup system config file

i. mailto: [email protected] e. Select and click “Open” and this initiates the install.

1. Contact Name f. Wait and when it is finished it will tell you that it was successful.

2. Company Name Errors

3. SG serial number and operating system version. Errors are reported for all types of reasons, but getting errors doesn’t
2. Backup the “configuration-passwords-key” necessarily mean the install wasn’t successful. Some errors may be
(see: “Export- Import SSL Keys”) expected. If errors are generated whilst restoring the archive; be sure to
save them (copy/paste to a text file) so that they can be examined later,
3. Backup non-default SSL keys (see: “Export- Import SSL Keys”)
in the event something does not function as expected. The errors may
4. Backup the system config via the Management GUI help understand the cause.
a. Configuration > General > Archive > View File: Examples of expected errors:
b. Select “Configuration – expanded”
1. Restoring the config before restoring the “configurations-passwords-
i. If the archive will be restored to multiple systems then “post key”.
setup” is the better archive option. For more information see
“Archive Types” at the end of this document. a. This will generate errors as the proxy tries to decrypt configured
passwords and fails. However, the config will be restored
c. Click “View” (brings up the config in a new browser window) successfully it just means those passwords will need to be set
SUPPORT PARTNER d. Save using your browsers FILE | SAVE function and save as TEXT manually from the management console.
TROUBLESHOOTING GUIDE
2. Restoring a config without first installing the content filter database.

52
 USER GUIDE
a. The archive may contain policy referencing content filter categories Archive Using the CLI
that do not yet exist. The configuration will be successfully restored, SGOS>en
but the policy will not install until the content filter database has Enable Password:
been downloaded. SGOS #config t
3. Restoring network settings to an SG that already has network SGOS #(config)archive-configuration host x.x.x.x
information configured. SGOS #(config)archive-configuration protocol ftp
SGOS #(config)archive-configuration path //bluecoat/
a. This will generate errors during the restore, however the config
configuration will be restored successfully using the existing SGOS #(config)archive-configuration username
network settings. <username>
The key to knowing whether or not the restore was successful is to test SGOS #(config)archive-configuration password
to be sure that the proxy is functioning as expected. <password>
SGOS #(config)exit
NOTE: Always check to be sure you “Default Proxy Policy” is set to what
SGOS #upload configuration
you intend (Allow/Deny).
% Uploading ftp://x.x.x.x/bluecoat/config
Configuration > Policy > Policy Options ok
SGOS #

FTP protocol used to upload the configuration (ftp or tftp is available)

X.X.X.X IP of the FTP server where you want to upload the configuration.
BLUECOAT/CONFIG path off of the root directory on the ftp server to upload the
configuration to
<USERNAME> user name used to log into the ftp server
<PASSWORD> password used to log into the ftp server

Restoring the Configuration File Using the CLI

To restore a configuration file onto a ProxySG using the command line
interface the file needs to be located on an FTP server that is accessible
by the proxy.
• Copy the configuration file from onto an FTP server if it is not on one
already
• Using an FTP browser, locate the archived configuration to be
restored and note the URL.
SUPPORT PARTNER • Access the serial console of the proxy using SSH or direct serial
TROUBLESHOOTING GUIDE connection

53
 USER GUIDE
• Enter “enable” mode on the proxy 6. RADIUS secrets
#enable 7. LDAP search password
• Enter enable password 8. Content Filter download passwords
• At the enable command prompt, enter the following command: 9. SNMP read, write, and trap community strings
SGOS#configure network “<URL noted earlier>”
10. Etc, etc…
This will work if the FTP supports anonymous login.
An SG’s archived configuration contains passwords which have all been
For example encrypted with the existing configuration-passwords-key. Once the
• #configure network ftp://x.x.x.x/archived-file-name. SG has been restored to a new state (re-initialization on a single disk
config system, restore to factory defaults, RMA, etc..) this key will be recreated,
but it will not be the same key that was used to encrypt the existing
If the above statement does not work try entering the URL in quotes
passwords.
#configure network “ftp://x.x.x.x/archived-file-
Upon restoring the archive the SG will attempt to decrypt the passwords
name.config”
using its new configuration-passwords-key. Since this is not the key
• If the FTP server requires username and password the username and used to encrypt the passwords originally this process will fail (see ftp-
password can be embedded into the URL. The format of the URL is: client example below).
ftp://username:password@ftp-server where ftp-server is either the IP
address or the DNS resolvable hostname of the FTP server.
#configure network ftp://username:password@ftp-
server/archived-file-name.config

Export-Import SSL Keys

The process discussed here is used for all SSL keys, but this section
uses the “configuration-passwords-key” as an example as this is one
of the SSL keys created on the SG by default and requires some extra
explanation.
The “configuration-passwords-key” is used to encrypt the various
passwords stored for use on the SG.
Examples of Encrypted passwords:
1. Administrator console passwords
2. Privileged-mode (enable) passwords While the passwords can be manually reset one by one via the
3. The front-panel PIN management console after the configuration has been restored it is
SUPPORT PARTNER possible to avoid this step by simply exporting the existing key and
4. Failover group secret importing it again before restoring the configuration to the SG.
TROUBLESHOOTING GUIDE
5. Access log FTP client passwords (primary, alternate)

54
 USER GUIDE
Export the SSL key Import the SSL Key
1. Enter enable mode and configuration terminal (config t) 1. Launch the Management Console
2. ssl 2. Configuration>SSL>Keyrings
3. view keyring 3. configuration-passwords-key
a. listing of keyrings is displayed a. delete the existing key (click apply)
4. view keypair des3 configuration-passwords-key b. create a new one using the exported key
a. enter an encryption password when prompted (remember this i. Keyring Name: configuration-passwords-key
password…it will be used later to restore the key) ii. Select “Show keypair”

5. Copy and paste the private key to a text file. iii. Leave the default “1024” –bit keyring
iv. Click “Import keyring
Example:
v. Paste configuration-passwords-key, then select “Paste From
Clipboard” button (characters will be masked, to view Tick the
box for “Show in Plain Text”)
vi. Enter the password used to encrypt it in “Step 4a” above.
vii. Click “OK”
viii. Click “Apply”
4. The configuration-passwords-key is now successfully imported.

Below is the portion of the key that must be copied and pasted to a text
file (including the BEGIN and END RSA PRIVATE KEY lines).

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

55
 USER GUIDE
Reinitializing the Disk(s) on the ProxySG d. Type:

There may be times when a disk re-initialization is necessary to correct i. disk reinitialize 1
certain problems. 1. Wait until it finishes

Single Disk System 2. Check the event log for disk related errors and save as
Disk re-initialization is not permitted on single disk systems, attempting “reinit-1.log”
to run the CLI command will generate the following error. a. https://<x.x.x.x>:8082/eventlog/fetch=0xFFFFFFFF
b. save the log using the browser save function
ii. disk reinitialize 2
1. Wait until it finishes
To clear the disk on a single system you have to follow the same
process as Factory Restoring the device to default settings. 2. Check the event log for disk related errors and save as
“reinit-2.log”
Instructions on how to do this can found in another section further in this
appendix. a. https://<x.x.x.x>:8082/eventlog/fetch=0xFFFFFFFF

Multiple Disk System b. Save the log using the browser save function

It is much simpler to reinitialize a multiple disk system. This process can e. Upload the event logs: upload.bluecoat.com
be done during production hours without impact to production, however,
unless absolutely necessary it is best to perform a reinitialize operation
during low traffic or off hours. When traffic is at a minimum and therefor
Disk function is reduced.
1. Backup the system configuration and SSL keys (Appendix G) The process for reinitializing is normally quick and problem free. Each
2. Reinitialize the disks one at a time disk on the SG contains a copy of the system configuration (mirrored).
When a disk is reinitialized it is taken off-line. When this happens the
a. Use the “slot number” for reference when reinitializing disks. Start
ProxySG will automatically use the next active disk as primary and
with slot 1 and work up. Be sure to look over the “Sysinfo” file to
continue processing traffic. This continues until all disks have been
review what the actual slot numbers are.
reinitialized.
i. https://<x.x.x.x>:8082/sysinfo
In the event a problem does occur and for some reason the
ii. Under: “Hardware Information” configuration is lost during a reinitialize then it will be necessary to
restore the system configuration from backup (Appendix G).
b. Launch the CLI (ssh or serial)
After re-initialization the current configuration will be saved back to this
c. Enable mode
disk from one of the other active disks.

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

56
 USER GUIDE
Factory Restoring the Device
This can be performed on ALL SG devices, but with single disk systems
is the only method available to re-initialize the disk.
There are 3 methods of restoring system defaults.
1. Restoring Factory Defaults
This option will reset all system settings, including the Setup Console
password. To restore factory defaults, use the following CLI command:
SGOS# restore-defaults factory-defaults
**This command should be used on Single Disk Systems in order to
re-initialize the disk
2. Retaining Console Settings
This option will retain settings like console username and password,
front panel pin number, console enable password, SSH host keys,
keyrings used by secure console services, RIP configurations, IP
address, MTU size, TCP round trip time, and static routes. To restore
defaults while retaining console settings, use the following CLI
command:
SGOS# restore-defaults keep-console
3. Restoring Defaults through the Management Console
In the Management Console, go to Maintenance > Systems and Disks >
Tasks
Click the Restore button. This will send the restore -defaults keep-
console command to the proxy.
Retained settings:
• IP Addresses, including Gateways and DNS (VIPs are not retained)
• Settings for all consoles
• Static Routes information

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

57
 USER GUIDE
APPENDIX H: PROXYAV LOGS.ZIP FILE
Logs.zip File 3. Alternatively you can upload the logs.zip directly into an SR via the
“Send Service Information” option on the same management console
(https://kb.bluecoat.com/index?page=content&id=FAQ1369)
screen as, see below. Simply enter the SR number and click “Send
This is an archive file generated by the Proxy AV upon request which button.
contains ALL the debug data that the AV device produces.
It can be created via the following methods
1. Browse to the advance URL of https://x.x.x.x:8082/logs.zip
2. Browse to the Management GUI of the device. Choose “Advanced”
then “Troubleshooting”. Then select “click here to download
troubleshooting file”.

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

58
 USER GUIDE
APPENDIX I: HOW TO CREATE A PACKET CAPTURE ON THE PROXYAV
How to create a packet capture file on the Proxy AV Similar to that seen below
(https://kb.bluecoat.com/index?page=content&id=FAQ1370&actp=LIST)
1. Browse to the Management GUI of the Proxy AV
a. Utilities
b. Diagnostics
2. Leave Filtering Condition IP addresses blank unless requested
otherwise by Blue Coat Support
3. Tick boxes labeled
a. “Enable Connection Data Logging”
b. “Enable Packet Logging”
4. Configure a Time Period that the packet capture should run for. Enter
high value to ensure PCAP doesn’t stop during replication test.
5. Click Start button
6. Replicate the issue
7. Click Stop button
8. PCAP file will be saved as in the following format
‘PacketLogYYMMDD-hhmmss.log’
9. Download the PCAP file from the list located on the same page, under
“Logs”
10. Upload to SR

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

59
 USER GUIDE
APPENDIX J: TAKING A PACKET CAPTURE ON DIRECTOR
Use standard Linux tcpdump CLI commands to take a packet capture
from Director. By default tcpdump only captures the first 68 bytes. In
order to retain the header information a filter must be set using the –s
(snaplen) option before starting the packet capture.
From the Director CLI (https://<directorIPaddr>:8082):
Log in
Enable
Conf t
Getting a packet capture:
1. (config) # tcpdump filter -s200
a. This sets the number of bytes to capture per packet, 0 captures full
packet (-s0)
2. (config) # tcpdump start
3. duplicate the issue
4. (config) # tcpdump stop
5. Upload the PCAP from Director to FTP or HTTP server (file name:
“sgmetcpdump” or “directortcpdump”)
a. (config) # tcpdump upload ftp://<hostname>/<path>
NOTE: Either of these two options can be used...
http://<hostname[:port]>/<path>
ftp://<hostname>/<path>
If <path> ends with a directory name, it must end with /
(ex): tcpdump upload ftp://192.168.1.251/mark/
6. Upload the packet capture to the case by using
https://upload.bluecoat.com

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

60
 USER GUIDE
APPENDIX K: TAKING A DEBUG DUMP ON DIRECTOR
The Debug Dump file should always be generated AFTER the issue
occurs, it can be done so using the following commands.
1. Immediately after reproducing the issue go to the director CLI (https://
<directorIPaddr>:8082)
2. Log in
3. Enable
4. Conf t
(config)#debug dump generate
Generating debugging dump...
Dump file successfully written to
ciqinfo-Director-2007.06.01-155844.tgz
5. Upload the dump to a local FTP server
(config) # shell
sh-2.05b# cd /local/userfiles
sh-2.05b# ls
cIQconfig_050202dump?ciqinfo-
Director-2007.06.01-155844.tgz
sh-2.05b# mv dump?ciqinfo-
Director-2007.06.01-155844.tgz Debug_dump.tgz
sh-2.05b#ftp
ftp> open ftp.example.com
ftp> bin
ftp> put Debug_dump.tgz

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

61
 USER GUIDE
APPENDIX L: DIRECTOR MESSAGES FILE (AUDIT LOGGING)
Audit Logging tracks the contents of Profiles, Overlays, Configuration Remote system type is UNIX.
and content jobs, backups when executed. As well as the Username ftp> put messages
and IP address of the user who executed them. local: messages remote: messages
227 Entering Passive Mode (10,91,1,59,227,172)
To view the username information TACACS+ authentication must be
150 Connection accepted
configured for management access to the Director device, otherwise this
226 Transfer OK
information will not be logged.
50877353 bytes sent in 5.15 secs (9.6e+03 Kbytes/
The messages file can be found in the linux shell of the device (/var/log/ sec)
messages), it is necessary to set the Local Logging level to “Debug”
4. Change the Local Logging level back to “Notice” using the following
BEFORE replicating ALL issues, in order to capture more granular
command
messages at the time the problem occurs.
director (config) # logging local notice
**It is VERY important to set the Local Logging level back to “Notice”
AFTER replication, otherwise the messages file will fill up quickly and The Audit Logging Files are stored in subdirectories of /local/logs/
use up space on the hard disk which can cause issues.** scplogs (for example, the contents of backup jobs are stored in /local/
logs/scplogs/backups).
1. Configure the Local Logging Level to “Debug” using the following CLI
command • Event logs, stored in the /var/log/messages file, are transferred every
hour to the /local/logs/scplogs/messages directory using a cron job.
director (config) # logging local debug
• A cron job runs every five minutes to transfer audit logs from
2. Replicate the issue
subdirectories of local/logs/scplogs to an external server using the
3. Download the messages file from Director AFTER replication. SCP, if a server is configured.
The example below is uploading via FTP to an FTP server on the
• After the files are transferred, the logs are deleted; however, if no
local network, which requires authentication, using the FTP “PUT”
external server is specified, no transfer takes place. After the contents
command to upload the file.
of the audit log directory reach
director # config t
1GB in size, the overflow policy is enacted. The overflow policy can be
director (config) # shell
set to delete the oldest log files first (the default), to disable commands
Releasing the configuration lock.
that trigger audit logging, or to stop creating new audit log files.
sh-3.2# cd /var/log
sh-3.2# ftp x.x.x.x Generally Event Logging output is required by Blue Coat Support for
Connected to x.x.x.x (x.x.x.x). troubleshooting; however they may require the Audit Logging output on
220-FileZilla Server version 0.9.41 beta occasion.
220-written by Tim Kosse ([email protected])
To Manually Clear the Audit Logging Files use the following commands
220 Please visit http://sourceforge.net/projects/
filezilla/ (config) # logging dump-contents clear
Name (x.x.x.x:admin): username
SUPPORT PARTNER 331 Password required for username
TROUBLESHOOTING GUIDE Password:
230 Logged on
62
 USER GUIDE
APPENDIX M: REPORTER DIAGNOSTICS FILE
To upload a Reporter Diagnostics File Directly from Reporter GUI.
Browse to the Administration Tab, then System Diagnostics and enter
the Service Request number in the “SR Number” field and press
“Upload.”
If the Reporter GUI does not load, or the Reporter service does not start
you can still generate a Reporter Diagnostics File.
Navigate to the root of your Reporter installation.
For Windows:
• “C:\Program Files\Blue Coat Reporter 9\”
For Linux:
• /opt/bc/reporter/
Run “bcrdiagnostics” (it will have an .exe extension in Windows), it will
prompt for your SR number and press return.
If your server does not have direct Internet access, the upload will fail,
but the zip file will be generated in the root install directory regardless.
This can then be uploaded via https://upload.bluecoat.com

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

63
 USER GUIDE
APPENDIX N: REPORTER JOURNAL FILE
You can find those Journal files in the following default locations on your
Reporter server:
Windows: C:\Program Files\Blue Coat Reporter 9\journal
Linux: /opt/bc/reporter/journal

SUPPORT PARTNER
TROUBLESHOOTING GUIDE

64
 WHITEPAPER
About Symantec

Symantec Corporation (NASDAQ: SYMC), the world’s leading cyber security company, helps
organizations, governments and people secure their most important data wherever it lives.
Organizations across the world look to Symantec for strategic, integrated solutions to defend against
sophisticated attacks across endpoints, cloud and infrastructure. Likewise, a global community of more
than 50 million people and families rely on Symantec’s Norton suite of products for protection at home
and across all of their devices. Symantec operates one of the world’s largest civilian cyber intelligence
networks, allowing it to see and protect against the most advanced threats. For additional information,
please visit www.symantec.com or connect with us on Facebook, Twitter, and LinkedIn.

Symantec Corporation World Headquarters

350 Ellis Street
Mountain View, CA 94043 USA
+1 (650) 527 8000
1 (800) 721 3934

www.symantec.com

Copyright © 2017 Symantec Corporation. All rights reserved. Symantec, the Symantec Logo, and the Checkmark Logo are trademarks or registered trademarks of Symantec
Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. 65