Veritas NetBackup™ Logging Reference Guide - NetBackup81 - Logging - RefGd
Veritas NetBackup™ Logging Reference Guide - NetBackup81 - Logging - RefGd
Release 8.1.1
Veritas NetBackup™ Logging Reference Guide
Legal Notice
Copyright © 2018 Veritas Technologies LLC. All rights reserved.
Veritas, the Veritas Logo, and NetBackup are trademarks or registered trademarks of Veritas
Technologies LLC or its affiliates in the U.S. and other countries. Other names may be
trademarks of their respective owners.
This product may contain third party software for which Veritas is required to provide attribution
to the third party (“Third Party Programs”). Some of the Third Party Programs are available
under open source or free software licenses. The License Agreement accompanying the
Software does not alter any rights or obligations you may have under those open source or
free software licenses. Refer to the third party legal notices document accompanying this
Veritas product or available at:
https://www.veritas.com/about/legal/license-agreements
The product described in this document is distributed under licenses restricting its use, copying,
distribution, and decompilation/reverse engineering. No part of this document may be
reproduced in any form by any means without prior written authorization of Veritas Technologies
LLC and its licensors, if any.
http://www.veritas.com
Technical Support
Technical Support maintains support centers globally. All support services will be delivered
in accordance with your support agreement and the then-current enterprise technical support
policies. For information about our support offerings and how to contact Technical Support,
visit our website:
https://www.veritas.com/support
You can manage your Veritas account information at the following URL:
https://my.veritas.com
If you have questions regarding an existing support agreement, please email the support
agreement administration team for your region as follows:
Japan [email protected]
Documentation
The latest documentation is available on the Veritas website:
https://sort.veritas.com/documents
Documentation feedback
Your feedback is important to us. Suggest improvements or report errors or omissions to the
documentation. Include the document title, document version, chapter title, and section title
of the text on which you are reporting. Send feedback to:
[email protected]
You can also see documentation information or ask a question on the Veritas community site:
http://www.veritas.com/community/
■ About logs
About logs
NetBackup uses several different logs and reports to help you troubleshoot any
problems that you encounter.
Users need to know where the log and report information is on their systems.
Figure 1-1 shows the location of the log and report information on the client and
the server and the processes that make the information available.
Using logs 9
About logs
NetBackup
Error
Database
Catalog Client
Manager Progress
Debug
Files
Logs
Media
NetBackup Catalog
Administrative
Interface
You can review a functional overview that describes the programs and daemons
that are mentioned in this figure.
You can also use NetBackup reports to help troubleshoot problems. NetBackup
reports give information about status and errors. To run reports, use the NetBackup
Administration Console.
See the Reports information in the NetBackup Administrator’s Guide, Volume I.
Note: The log-entry format in the NetBackup logs is subject to change without
notice.
Using logs 10
About UNIX system logs
Note: On HP-UX, the sysdiag tool may provide additional information on hardware
errors.
To enable additional logging by NetBackup to the system logs, use one of the
following:
■ Use the ltid command that started the device management processes. If the
-v option is included on the ltid command, all daemons that were started as
a result also have the -v option in effect.
■ Use a command to start a specific daemon (for example, acsd -v).
On UNIX, enable debug logging to the system logs by including the verbose option
(-v) on the command that you use to start a daemon.
To troubleshoot ltid or robotic software, you must enable system logging. See the
syslogd(8) man page for information on setting up system logs. Errors are logged
with LOG_ERR, warnings with LOG_WARNING, and debug information with LOG_NOTICE.
The facility type is daemon.
See the syslogd man page for the locations of system log messages on your
system.
Note: You can verify the log pruning behavior in NetBackup by using the logs at
the following location:
On Windows: install_path\NetBackup\logs\nbutils
On UNIX: /usr/openv/netbackup/logs/nbutils
Keep logs up to GB To limit the size of unified and legacy See “About limiting the size of unified and legacy
logs. logs” on page 12.
NumberOfLogFiles To limit the number of unified log files that See “About recycling unified log files”
you want to retain for a NetBackup on page 25.
process.
MaxLogFileSizeKB and To prevent unified log files from becoming See “About rolling over unified log files”
other vxlogcfg options too large. on page 24.
When a file size or time setting is
reached, the current log file is closed.
New log messages for the logging
process are written or “rolled over” to a
new log file.
Keep logs for days To limit the days for which NetBackup See “About limiting the size and the retention of
retains legacy logs. legacy logs” on page 47.
MAX_LOGFILE_SIZE To limit the legacy log size and the See “Configuring the legacy log rotation”
and number of legacy log files to be retained. on page 49.
MAX_NUM_LOGFILES
These options can be set using a
command-line interface.
Note: Before you enable logging for critical NetBackup processes, review the log
retention options and select them appropriately.
Note: You can verify the log pruning behavior in NetBackup by creating the following
directories:
On Windows: install_path\NetBackup\logs\nbutils
On UNIX: /usr/openv/netbackup/logs/nbutils
Using logs 13
About unified logging
Windows install_path\NetBackup\logs
UNIX /usr/openv/logs
vxlogmgr Manages the log files that the products that support unified logging
generate.
Windows install_path\NetBackup\bin
UNIX /usr/openv/netbackup/bin
Using logs 14
About unified logging
See the NetBackup Commands Reference Guide for a complete description about
these commands.
More information about legacy logging is available.
See “About legacy logging” on page 36.
# mkdir /upload
2 Copy unified logs (for NetBackup only) to the /upload directory by using the
following command:
Example output:
# cd /upload
ls
Example output:
51216-111-2202872032-050125-0000000.log
51216-116-2202872032-050125-0000000.log
51216-117-2202872032-050125-0000000.log
51216-118-2202872032-050125-0000000.log
51216-132-2202872032-050125-0000000.log
51216-157-2202872032-050125-0000000.log
Application log Application log messages include informational, warning, and error
messages messages. They are always logged and cannot be disabled. These
messages are localized.
An example of an application message follows:
Diagnostic log Diagnostic log messages are the unified logging equivalent of the legacy
messages debug log messages. They can be issued at various levels of detail
(similar to verbose levels in legacy logging). These messages are
localized.
Debug log Debug log messages are intended primarily for Veritas engineering.
messages Like diagnostic messages, they can be issued at various levels of detail.
These messages are not localized.
/usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000000.log
Table 1-2 Description of the file name format for unified logging
51216 Product ID Identifies the product. The NetBackup product ID is 51216. The
product ID is also known as the entity ID.
116 Originator ID Identifies the log writing entity, such as a process, service,
script, or other software. The number 116 is the originator ID
of the nbpem process (the NetBackup policy execution
manager).
2201360136 Host ID Identifies the host that created the log file. Unless the file was
moved, this ID is the host where the log resides.
041029 Date Shows the date when the log was written in YYMMDD format.
0000000000 Rotation Identifies the numbered instance of a log file for a given
originator. The rollover number (rotation) indicates the instance
of this log file. By default, log files roll over (rotate) based on
file size. If the file reaches maximum size and a new log file is
created for this originator, the new file is designated
0000000001.
The log configuration file specifies the name of the directories where the log files
for originator IDs are written. These directories and the log files that they hold are
written to the following directory, except as noted in the following:
See “Originator IDs for the entities that use unified logging” on page 17.
Windows install_path\NetBackup\logs
UNIX /usr/openv/logs
Table 1-3 Originator IDs for the server entities that use unified logging
103 pbx_exchange The Private Branch Exchange (PBX) service provides single-port access
to clients outside the firewall that connect to Veritas product services.
Service name: VRTSpbx. It writes logs to /opt/VRTSpbx/log (UNIX)
or install_path\VxPBX\log (Windows). The PBX product ID is
50936.
111 nbemm The Enterprise Media Manager (EMM) is a NetBackup service that
manages the device and the media information for NetBackup. It runs
only on the master server.
Using logs 18
About unified logging
Table 1-3 Originator IDs for the server entities that use unified logging
(continued)
116 nbpem The NetBackup Policy Execution Manager (nbpem) creates policy and
client tasks and determines when jobs are due to run. It runs only on the
master server.
117 nbjm The NetBackup Job Manager (nbjm) accepts the jobs that the Policy
Execution Manager submits and acquires the necessary resources. It
runs only on the master server.
118 nbrb The NetBackup Resource Broker (nbrb) maintains a cache list of
available resources. It uses that list to locate the physical and the logical
resources that are required for a backup or a tape restore. It initiates a
SQL call to nbemm to update the database, and then passes the allocation
information to nbjm. It runs only on the master server.
119 bmrd The NetBackup Bare Metal Restore (BMR) master server daemon.
121 bmrsavecfg The BMR Save Configuration is a data collection utility that runs on the
NetBackup client, not the server.
122 bmrc The BMR Client Utility originates on the BMR boot server and runs on
the restoring client. UNIX clients use it to communicate to the BMR
master server during a restore.
124 bmrcreatefloppy The BMR commands that create floppy disks use the BMR Create Floppy
utility. The utility runs on the BMR boot server and is Windows only.
125 bmrsrt The BMR Create SRT utility creates a shared resource tree. It runs on
the BMR boot server.
126 bmrprep The BMR Prepare to Restore utility prepares the BMR servers for a client
restoration.
127 bmrsetup The BMR Setup Commands utility sets up BMR installation, configuration,
and upgrade processes.
128 bmrcommon The BMR Libraries and Common Code catalog provides log messages
to the BMR libraries.
129 bmrconfig The BMR Edit Configuration utility modifies the client configuration.
130 bmrcreatepkg The BMR Create Package utility adds Windows drivers, service packs,
and hotfixes to the BMR master server for restore operations.
Using logs 19
About unified logging
Table 1-3 Originator IDs for the server entities that use unified logging
(continued)
131 bmrrst The BMR Restore utility restores Windows BMR clients. It runs on the
restoring client for Windows systems only.
132 nbsl The NetBackup Service Layer facilitates the communication between
the NetBackup graphical user interface and NetBackup logic. nbsl is
required to run NetBackup OpsCenter, an application that manages and
monitors multiple NetBackup environments. This process runs only on
the master server.
134 ndmpagent The NDMP agent daemon manages NDMP backups and restores. It
runs on the media server.
137 libraries The libraries control the logging level in the NetBackup libraries. The
application and the diagnostic messages are for customer use; the debug
messages are intended for Veritas engineering.
140 mmui The media server user interface is used for the Enterprise Media Manager
(EMM).
142 bmrepadm The BMR External Procedure process manages the BMR external
procedures that are used during a restore operation.
143 mds The EMM Media and Device Selection process manages the media
selection component and device selection component of the Enterprise
Media Manager (EMM).
151 ndmp The NDMP message log (ndmp) handles NDMP protocol messages,
avrd, and robotic processes.
154 bmrovradm The BMR Override Table Admin Utility manages the custom override
functions for Bare Metal Restore.
Using logs 20
About unified logging
Table 1-3 Originator IDs for the server entities that use unified logging
(continued)
156 ace The NBACE process controls the logging level in the (ACE/TAO) CORBA
components for any process that uses a CORBA interface. The default
level is 0 (only important messages are logged). This logging is intended
for Veritas engineering.
If Veritas Technical Support instructs you to increase the logging level,
increase the level for originator ID 137 to 4 or higher.
Warning: A debug logging level greater than 0 generates large amounts
of data.
163 nbsvcmon The NetBackup Service Monitor monitors the NetBackup services that
run on the local computer and tries to restart a service that unexpectedly
terminates.
166 nbvault The NetBackup Vault Manager manages NetBackup Vault. nbvault
must be running on the NetBackup Vault server during all NetBackup
Vault operations.
178 dsm The Disk Service Manager (DSM) performs set and get operations on
disk storage and disk storage units.
199 nbftsrvr The Fibre Transport (FT) server process runs on the media servers that
are configured for the NetBackup Fibre Transport. On the server side of
the FT connection, nbftsrvrcontrols data flow, processes SCSI
commands, manages data buffers, and manages the target mode driver
for the host bus adapters. nbftsrvr is part of SAN client.
200 nbftclnt The Fibre Transport (FT) client process runs on the client and is part of
SAN Client.
201 fsm The FT Service Manager (FSM) is a component of the Enterprise Media
Manager (EMM) and is part of SAN Client.
202 stssvc The Storage service manages the storage server and runs on the media
server.
Table 1-3 Originator IDs for the server entities that use unified logging
(continued)
219 rsrcevtmgr The Resource Event Manager (REM) is a CORBA loadable service that
runs inside nbemm. REM works with the Disk Polling Service to monitor
free space and volume status, and to watch for disk-full conditions.
221 mpms The Media Performance Monitor Service (MPMS) runs on every media
server within RMMS and gathers CPU load and free memory information
for the host.
222 nbrmms Remote monitoring and Management Service (RMMS) is the conduit
through which EMM discovers and configures disk storage on media
servers.
226 nbstserv The Storage services controls the lifecycle image duplication operations.
230 rdsm The Remote Disk Service Manager interface (RDSM) runs within the
Remote Manager and Monitor Service. RDMS runs on media servers.
231 nbevtmgr The Event Manager Service provides asynchronous event management
services for cooperating participants.
248 bmrlauncher The BMR Launcher Utility in the Windows BMR Fast Restore image
configures the BMR environment.
254 SPSV2RecoveryAsst Recovery Assistant for SharePoint Portal Server for NetBackup clients.
272 expmgr The Expiration Manager handles the capacity management and the
image expiration for storage lifecycle operations.
Table 1-3 Originator IDs for the server entities that use unified logging
(continued)
Table 1-3 Originator IDs for the server entities that use unified logging
(continued)
387 nbrntd NetBackup Remote Network Transport. If multiple backup streams run
concurrently, the Remote Network Transport Service writes a large
amount of information to the log files. In such a scenario, set the logging
level for OID 387 to 2 or less.
Table 1-4 vxlogcfg options that control the rollover of the unified log files
Option Description
MaxLogFileSizeKB Specifies the maximum size that is allowed for the log file (in
kilobytes) before rollover occurs, if the RolloverMode is set
to FileSize.
RolloverAtLocalTime Specifies the time of day at which the log file is rolled over,
if the RolloverMode is set to LocalTime.
RolloverPeriodInSeconds Specifies a period of time in seconds after which the log file
is rolled over, if the RolloverMode is set to Periodic.
MaxLogFileSizeKB or Specifies that the log file rollover occurs whenever the file
RolloverAtLocalTime size limit or the local time limit is reached, whichever is first.
MaxLogFileSizeKB or Specifies that the log file rollover occurs whenever the file
RolloverPeriodInSeconds size limit or the periodic time limit is reached, whichever is
first.
The previous example uses the vxlogcfg command with the RolloverMode option.
It sets rollover mode for nbpem (originator ID 116) to Periodic. It also sets the
interval until the next nbpem log file rollover to 24 hours (86400 seconds).
In the following example, the file names show the log file rollover with the rotation
ID incremented:
/usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000000.log
/usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000001.log
/usr/openv/logs/nbpem/51216-116-2201360136-041029-0000000002.log
In addition, you can use log file rotation with the following:
■ Logs for the server processes that use unified logging
See “Originator IDs for the entities that use unified logging” on page 17.
■ Certain legacy logs
■ The unified logging files that the Bare Metal Restore process bmrsavecfg creates
Limit the number of log Specify the maximum number of log files that NetBackup retains.
files When the number of log files exceeds the maximum, the oldest log
files become eligible for deletion during log cleanup. The
NumberOfLogFiles option for the vxlogcfg command defines
that number.
In the following example, the maximum number of log files that are
allowed for each of the unified logging originators in the NetBackup
(product ID 51216) is 8000. When the number of log files exceeds
8000 for a particular originator, the oldest log files become eligible
for deletion during log cleanup.
Specify the number of Use the Keep logs for days property to specify the maximum
days the log files are number of days logs are kept. When the maximum number of days
kept is reached, the unified logs and legacy logs are automatically
deleted.
In the NetBackup Administration Console, in the left pane, expand
NetBackup Management > Host Properties > Master Servers.
Double-click the server you want to change. A new dialog box
appears. In the left pane, click Logging > Keep logs for days.
Explicitly delete the log To initiate recycling and delete the log files, run the following
files command:
# vxlogmgr -a -d
If the vxlogcfg LogRecycle option is ON (true), the Keep logs for days setting
is disabled for unified logs. In this case, unified logging files are deleted when their
number (for a particular originator) exceeds the number that the NumberOfLogFiles
option specifies on the vxlogcfg command.
UNIX /usr/openv/logs
Windows install_path\NetBackup\logs
Unlike the files that are written in legacy logging, unified logging files cannot be
easily viewed with a text editor. The unified logging files are in binary format, and
some of the information is contained in an associated resource file. Only the
vxlogview command can assemble and display the log information correctly.
You can use vxlogview to view NetBackup log files as well as PBX log files.
To view PBX logs using the vxlogview command, do the following:
■ Ensure that you are an authorized user. For UNIX and Linux, you must have
root privileges. For Windows, you must have administrator privileges.
Using logs 27
About unified logging
■ To specify the PBX product ID, enter -p 50936 as a parameter on the vxlogview
command line.
vxlogview searches all the files, which can be a slow process. Refer to the following
topic for an example of how to display results faster by restricting the search to the
files of a specific process.
= equal to
!= not equal to
|| logical OR
Table 1-5 shows data types for specific fields as well as description and an example.
When more than one example is listed, both examples produce the same results.
Using logs 28
About unified logging
STDATE Long Integer or string Provide the start date in seconds or STDATE = 98736352
in the locale-specific short date and
STDATE = '4/26/11 11:01:00
time format. For example, a locale
AM'
can have the format 'mm/dd/yy
hh:mm:ss AM/PM'
ENDATE Long Integer or string Provide the end date in seconds or ENDATE = 99736352
in the locale-specific short date and
ENDATE = '04/27/11 10:01:00
time format. For example, a locale
AM'
can have the format 'mm/dd/yy
hh:mm:ss AM/PM'
1 = WARNING
2 = ERR
3 = CRIT
4 = EMERG
Using logs 29
About unified logging
Case sensitivity Field names, severity types, and message types are not case-sensitive.
For example, the following are valid entries:
■ sev = info
■ msgtype = diag
String constants String constants should be given in single quotes. For example, PRODID
= 'NBU'
Dates Start and end dates can be provided in the following formats:
■ A string constant that corresponds to the regional display short date
format
■ A UNIX long value of number of seconds that elapsed since midnight
January 1, 1970.
Example Description
(PRODID == 51216) && ((PID == 178964) Retrieves the log file message for the
|| ((STDATE == '2/5/15 09:00:00 AM') NetBackup product ID 51216 between
&& (ENDATE == '2/5/15 12:00:00 PM')) 9AM and 12PM on 2015-05-02.
Using logs 30
About unified logging
Example Description
((prodid = 'NBU') && ((stdate >= Retrieves the log messages for the
‘11/18/14 00:00:00 AM’) && (endate NetBackup product NBU between
<= ‘12/13/14 12:00:00 PM’))) || 2014-18-11 and 2014-13-12 and the log
((prodid = 'BENT') && ((stdate >= messages for the NetBackup product
‘12/12/14 00:00:00 AM’) && BENT between 2014-12-12 and
(endate <= ‘12/25/14 12:00:00 PM’))) 2014-25-12.
(STDATE <= ‘04/05/15 0:0:0 AM’) Retrieves the log messages that were
logged on or before 2015-05-04 for all
of the installed Veritas products.
Item Example
Display specific Display the log messages for NetBackup (51216) that show only
attributes of the log the date, time, message type, and message text:
messages
vxlogview --prodid 51216 --display D,T,m,x
Display the latest log Display the log messages for originator 116 (nbpem) that were
messages issued during the last 20 minutes. Note that you can specify -o
nbpem instead of -o 116:
Display the log Display the log messages for nbpem that were issued during the
messages from a specified time period:
specific time period
# vxlogview -o nbpem -b "05/03/15 06:51:48 AM"
-e "05/03/15 06:52:48 AM"
Using logs 31
About unified logging
Item Example
Display results faster You can use the -i option to specify an originator for a process:
# vxlogview -i nbpem
The vxlogview -i option searches only the log files that the
specified process (nbpem) creates. By limiting the log files that it
has to search, vxlogview returns a result faster. By comparison,
the vxlogview -o option searches all unified log files for the
messages that the specified process has logged.
Note: If you use the -i option with a process that is not a service,
vxlogview returns the message "No log files found." A process
that is not a service has no originator ID in the file name. In this
case, use the -o option instead of the -i option.
The -i option displays entries for all OIDs that are part of that
process including libraries (137, 156, 309, etc.).
Search for a job ID You can search the logs for a particular job ID:
See the NetBackup Commands Reference Guide for a complete description of the
vxlogview command. The guide is available through the following URL:
http://www.veritas.com/docs/DOC5332
Item Example
List the log files List all unified log files for the nbrb service:
# vxlogmgr -s -o nbrb
/usr/openv/logs/nbrb/51216-118-1342895976-050503-00.log
/usr/openv/logs/nbrb/51216-118-1342895976-050504-00.log
/usr/openv/logs/nbrb/51216-118-1342895976-050505-00.log
Total 3 file(s)
Delete the oldest log If the vxlogcfg NumberOfLogFiles option is set to 1, the following example deletes the
files two oldest log files for the nbrb service:
Delete the newest log Delete all the unified log files that NetBackup created in the last 15 days:
files
# vxlogmgr -d --prodid 51216 -n 15
Make sure that you roll over (rotate) the log files before you recycle them.
Delete the log files for Delete all unified log files for originator nbrb:
a specific originator
# vxlogmgr -d -o nbrb
Make sure that you roll over (rotate) the log files before you recycle them.
Delete all the log files Delete all unified log files for NetBackup:
# vxlogmgr -d -p NB
Make sure that you roll over (rotate) the log files before you recycle them.
Using logs 33
About unified logging
Item Example
Control the number of You can use the vxlogmgr command with the vxlogcfg command’s NumberOfLogFiles
log files option to manually delete log files.
For example, the NumberOfLogFiles option is set to 2, you have 10 unified logging files,
and cleanup has not occurred. Enter the following to keep the two most recent log files and
delete the rest for all originators:
# vxlogmgr -a -d
The following command keeps the two most recent log files of all PBX originators:
# vxlogmgr -a -d -p ics
The following deletes the older log files for the nbrb service only:
# vxlogmgr -a -d -o nbrb
Control disk space Periodically run the vxlogmgr -a -d command (such as through a cron job) to delete
usage logs and monitor the disk space that unified logging uses.
The disk space that a given originator uses can be calculated as follows:
The total disk space that unified logs consume is the sum of the disk space that each originator
consumes. If none of the originators override the NumberOfFiles and MaxLogFileSizeKB
settings, then the total disk space that unified logging consumes is as follows:
Number of originators * default MaxLogFileSizeKB * default NumberOfFiles
Use the vxlogcfg command to list the current unified logging settings.
For example, assume the following:
To calculate the total disk space that unified logging consumes, insert the values from the
example into the previous formula. The results are as follows:
10 * 51200 * 3 KB = 1,536,000 KB of additional disk space used each hour.
Item Example
Set the maximum log By default, the maximum log file size in unified logging is 51200
file size KB. When a log file reaches 51200 KB, the file closes and a new
log file opens.
Item Example
Set log recycling The following example sets automatic log file deletion for nbemm
logs (originator ID 111):
This example sets nbemm rollover mode to file size, and turns on
log recycling. When the number of log files exceeds 999, the oldest
log file is deleted. EXAMPLE 5 shows how to control the number
of log files.
Set debug level and The following example sets the default debug level and diagnostic
diagnostic level level of product ID NetBackup (51216):
Item Example
List the unified logging The following vxlogcfg example shows how to list the active
settings unified logging settings for a given originator (the nbrb service).
Note that MaxLogFileSizeKB, NumberOfLogFiles, and
RolloverMode are included in the output.
# vxlogcfg -l -o nbrb -p NB
In legacy debug logging, each process creates log files of debug activity in its own
logging directory. The NetBackup legacy debug log directories are located in the
following directories:
Windows install_path\NetBackup\logs
install_path\Volmgr\debug
UNIX /usr/openv/netbackup/logs
/usr/openv/volmgr/debug
These top-level directories can contain a directory for each NetBackup process that
uses legacy logging. By default, NetBackup creates only a subset of all of the
possible log directories. For example, the following directories are created by default
on UNIX servers:
■ nbfp
■ nbliveup
■ nblogadm
■ user_ops
To enable logging for all of the NetBackup processes that use legacy logging, you
must create the log file directories that do not already exist, unless you use the
Logging Assistant. For more information about the Logging Assistant, see the
NetBackup Administrator's Guide, Volume I. The guide is available at the following
location:
http://www.veritas.com/docs/DOC5332
See “Directory names for legacy debug logs for servers ” on page 43.
See “Directory names for legacy debug logs for media and device management”
on page 45.
You can use the following batch files to create all of the debug log directories at
once:
■ Windows: install_path\NetBackup\Logs\mklogdir.bat
■ UNIX: usr/openv/netbackup/logs/mklogdir
See the NetBackup Commands Reference Guide for a complete description about
the mklogdir command. The guide is available at the following location:
http://www.veritas.com/docs/DOC5332
After the directories are created, NetBackup creates log files in the directory that
is associated with each process. A debug log file is created when the process
Using logs 38
About legacy logging
begins. Each log file grows to a certain size before the NetBackup process closes
it and creates a new log file.
See “File name format for legacy logging” on page 42.
To enable debug logging for the NetBackup Status Collection Daemon (vmscd),
create the following directory before you start nbemm.
Windows install_path\Volmgr\debug\vmscd\
UNIX /usr/openv/volmgr/debug/vmscd
You can use the following batch file to create all of the debug log directories at
once:
Windows Install_path\NetBackup\Logs\mklogdir.bat
UNIX usr/openv/netbackup/logs/mklogdir
Table 1-10 describes the directories for the legacy debug logs that apply to UNIX
clients.
Table 1-10 UNIX client processes that use legacy logging (continued)
bplist Program that lists backed up and archived files. Also useful to debug
bp. On pre-7.6 versions of NetBackup, the bpclntcmd command
and the bpclimagelist command send their debug log messages
to the bplist directory. On NetBackup 7.6, bpclntcmd and
bpclimagelist send their debug log messages to the bpclntcmd
and the bpclimagelist directory, respectively.
bpmount Program that determines the local mount points and wildcard
expansion for multiple data streams.
db_log For more information on these logs, see the NetBackup guide for the
database-extension product that you use.
mtfrd These logs have information about the mtfrd process that is used
for phase 2 imports and restores of the Backup Exec media.
Using logs 40
About legacy logging
Table 1-10 UNIX client processes that use legacy logging (continued)
Note: These are the default locations in which to place these directories. You can
specify another location during client installation.
Table 1-11 lists the legacy debug log directories that apply to these clients.
bpinetd All Windows clients Client service logs. These logs have
information on the bpinetd32
process.
bparchive All Windows clients Archive program that is run from the
command line.
Using logs 41
About legacy logging
bpbackup All Windows clients The backup program that is run from
the command line.
bplist All Windows clients List program that is run from the
command line.
bprestore All Windows clients The restore program that is run from
the command line.
In addition, on NetBackup
Java-capable platforms, the
NetBackup Java interface log files are
written in a subdirectory that is called
nbjlogs. All files in the user_ops
directory hierarchy are removed
according to the setting of the
KEEP_LOGS_DAYS configuration
option.
user_name.mmddyy_nnnnn.log
user_name The name of the user in whose context the process runs, as follows:
■ For UNIX root user, the user_name is root.
■ For UNIX user other than the root user, the user_name is the user's login
ID.
■ For all users who are part of the Administrator group in Windows, the
user_name is ALL_ADMINS.
■ For Windows user, the user_name is either username@domain_name
or username@machine_name.
Using logs 43
About legacy logging
mmddyy The month, day, and year on which NetBackup created the log file.
nnnnn The counter or the rotation number for the log file. When the counter exceeds
the setting for number of log files, the oldest log file is deleted.
The retention of all logs files in the legacy debug log directories is managed by
using the following options:
■ Keep logs for days setting of the NetBackup Host Properties Logging dialog
box. The default is 28 days.
■ Keep logs up to size setting of the NetBackup Host Properties Logging dialog
box.
■ The legacy logging settings.
See “About limiting the size and the retention of legacy logs” on page 47.
Any mixture of new and old log file names in a legacy debug log directory is managed
according to the Keep logs setting and the robust logging settings.
Windows install_path\NetBackup\logs
UNIX /usr/openv/netbackup/logs
bpcd NetBackup client daemon or manager. The NetBackup Client service starts this
process.
bpdbm NetBackup Database Manager. This process runs only on master servers. On
Windows systems, it is the NetBackup Database Manager service.
bpjava-msvc The NetBackup Java application server authentication service that is started when
the NetBackup interface applications start. On UNIX servers, inetd starts it. On
Windows servers, the Client Services service starts it.
bpjava-susvc The NetBackup program that bpjava-msvc starts upon successful logon through
the logon dialog box that is presented when a NetBackup interface starts. This
program services all requests from the Java user interfaces on the NetBackup master
or media server host where the bpjava-msvc program runs (all Windows platforms).
bprd NetBackup request daemon or manager. On Windows systems, this process is called
the NetBackup Request Manager service.
bpsynth The NetBackup process for synthetic backup. nbjm starts bpsynth. bpsynth runs
on the master server.
nbatd Authentication daemon (UNIX and Linux) or service (Windows). nbatd authenticates
access to interfaces of NetBackup services or daemons.
nbazd Authorization daemon (UNIX and Linux) or service (Windows). nbazd authorizes
access to interfaces of NetBackup services or daemons.
user_ops The user_ops directory is created during the install of NetBackup on all servers and
clients. NetBackup interface programs use it for the following: temporary files and
for job and progress log files that the Backup, Archive, and Restore program
(jbpSA) generates. This directory must exist for successful operation of any of the
Java programs and must have public read, write, and execute permissions. user_ops
contains a directory for every user that uses the Java programs.
In addition, on Java-capable platforms, the NetBackup Java interface log files are
written in the nbjlogs subdirectory. All files in the user_ops directory hierarchy
are removed according to the setting of the KEEP_LOGS_DAYS configuration option.
Using logs 45
About legacy logging
vnetd The Veritas network daemon, used to create firewall-friendly socket connections.
Started by the inetd(1M) process.
Note: Logging occurs in either the /usr/openv/logs directory or the
/usr/openv/netbackup/logs if the vnetd directory exists there. If the vnetd
directory exists in both locations, logging occurs only in
/usr/openv/netbackup/logs/vnetd.
More information is available on the programs and daemons that write the logs.
See “Multiplexed backup process” on page 67.
On UNIX systems, also refer to the README file in the /usr/openv/netbackup/logs
directory.
Directory names for legacy debug logs for media and device
management
The debug log directories enable logging for the media management processes
and device management processes. Table 1-13 describes the directories you need
to create to support legacy debug logs for media and device management. Each
directory corresponds to a process.
ltid Debug information on ltid, the Media Manager device daemon (UNIX),
or on the NetBackup Device Manager service (Windows), and on avrd.
Stop and restart ltid after creating the directory.
robots Debug information on all robotic daemons, which includes tldcd, tl8cd,
and tl4d daemons. Stop and restart robotic daemons.
Using logs 46
About legacy logging
Table 1-13 Media and device management legacy debug logs (continued)
vmscd Debug information for the NetBackup Status Collection daemon. Stop
and restart vmscd after creating the directory.
Unless it is noted, each directory should be created under the following directory.
Windows install_path\Volmgr\debug
UNIX /usr/openv/volmgr/debug
Warning: High verbose values can cause debug logs to become very large.
Windows install_path\Volmgr\
UNIX /usr/openv/volmgr/
Note: The following properties have been moved from the Clean-up host properties
to the Logging host properties: Keep logs and Keep Vault logs. On the Logging
properties screen, these properties are referred to as Keep logs for days and
Keep Vault logs for respectively.
See the NetBackup Administrator’s Guide, Volume I for more information about
logging properties.
To limit the amount of disk space that the logs consume, use robust logging. Robust
logging involves file rotation, like that which is used in unified logging. Robust logging
does not apply to media and device management logging.
See “About rolling over unified log files” on page 24.
Specify the maximum size for a log file and the maximum number of log files to
keep in a logging directory. When a log file grows to its maximum size, it closes
and a new file opens. If the number of log files exceeds the number that is allowed
for the directory, the oldest file is deleted.
Logs created by the following NetBackup processes can use log rotation (robust
logging):
■ bpbkar (UNIX/Linux client only)
■ bpbrm
■ bpcd
■ bpdbm
■ bpdm
■ bprd
■ bptm
■ nbproxy
For the logs that are created by the other NetBackup processes (except media and
device management logs), use the Keep logs for days property. The Keep logs
for days property may override the robust file logging settings. If Keep logs for
days is set to 10 days and robust file logging settings allow more than 10 days, the
logs are deleted on day 11.
For media and device management legacy logs, use the DAYS_TO_KEEP_LOGS setting
in the vm.conf file to control log file rotation. The default is 30 days. The vm.conf
file is located in the following directory:
Windows install_path\Volmgr\
Using logs 49
About legacy logging
UNIX /usr/openv/volmgr/
To retain logs for 3 days, enter the following in the vm.conf file:
DAYS_TO_KEEP_LOGS = 3
See the NetBackup Administrator’s Guide, Volume II for instructions about how to
use this entry.
Initially, the bp.conf file does not contain the MAX_LOGFILE_SIZE and
MAX_NUM_LOGFILES entries. In this case, the parameters are set to their default
values, which are 256 MB and infinite, respectively.
Note: Beginning in NetBackup 7.7, the robust logging option is enabled by default.
Using logs 50
About global logging levels
Windows install_pathNetBackup\bin\admincmd\
UNIX /usr/openv/netbackup/bin/admincmd/
Use the following UNIX example to set the maximum file size to 512 MB and
the maximum number of log files per log directory to 4:
#bpsetconfig
bpsetconfig> MAX_NUM_LOGFILES = 4
bpsetconfig>
CTRL-D
Minimum logging Includes very important, low-volume diagnostic messages and debug
messages.
The Host Properties Logging page or the Logging Assistant can set
minimum logging.
Legacy logs use the following values to represent minimum logging:
Disable logging The Host Properties Logging page or the Logging Assistant can set
disable logging.
Legacy logs use the following values to represent disabled logging:
■ UNIX: The bp.conf file displays VERBOSE=-2 (global) or
processname_VERBOSE = -2 for an individual process.
■ Windows: Registry displays the following hexadecimal value: 0xfffffffe
diagnostic messages or debug messages are logged. This level cannot be set
with the Global logging level list in the NetBackup Administration Console.
You can set it with the vxlogcfg command or the Logging Assistant.
See “Changing the logging level” on page 52.
See “Examples of using vxlogcfg to configure unified logs” on page 34.
■ A change to the Global logging level list affects the logging level of all
NetBackup and Enterprise Media Manager (EMM) processes on the server or
client. (The exceptions are PBX and media and device management logging.)
This setting overrides any previous settings.
■ If you make a change to the VERBOSE entry (or entries) in the bp.conf file or
entry in the vm.conf file, it only affects the legacy logging.
See “How to control the amount of information written to legacy logging files”
on page 46.
■ If you make a change with the vxlogcfg command, it only affects the unified
logging level.
A change to the Global logging level list does not affect the level of the following
logging processes:
■ PBX logging
See the NetBackup Troubleshooting Guide for more information on how to
access the PBX logs.
■ Media and device management logging (vmd, ltid, avrd, robotic daemons,
media manager commands)
See “Directory names for legacy debug logs for media and device management”
on page 45.
■ Any unified logging process whose debug level has been changed from the
default setting
3 Restart the daemons and services or run the command verbose option, if
available.
Using logs 54
Setting retention limits for logs on clients
■ To monitor legacy processes (process names that start with bp; for example,
bpdbm), configure the eventlog file.
Using logs 55
Logging options with the Windows Event Viewer
Note: For the settings in the vxlogcfg command or the eventlog file to take effect,
you must restart the NetBackup services.
and the following example message is written in the Windows Event Viewer
Application log when the operating system logging is enabled for nbrb:
Note: For this setting to take effect, you must restart the NetBackup services.
When you change this option, the ignorable error messages are also written to the
Windows Event Viewer Application log. For example, if you specify the following
command:
install_path\NetBackup\db\config\eventlog
56 255
Note: For this setting to take effect, you must restart the NetBackup services.
Using logs 56
Logging options with the Windows Event Viewer
The parameters in the eventlog represent severity and type. The parameters have
the following characteristics:
Both parameters are specified as decimal numbers and equate to a bitmap that
expresses the following values:
Severity 1 = Unknown
2 = Debug
4 = Info
8 = Warning
16 = Error
32 = Critical
Type 1 = Unknown
2 = General
4 = Backup
8 = Archive
16 = Retrieve
32 = Security
64 = Backup Status
You can configure the eventlog file to log the messages that include several
different severities and types. If you specify an entry of 56 255 in the eventlog file,
the results are as follows:
Entry 56 Produces a log with the messages that have a severity of warning, error,
and critical. (56 = 8 + 16 + 32)
Using logs 57
Troubleshooting error messages in the NetBackup Administration Console
Entry 255 Produces a log with messages for all types. (255 = 1 + 2 + 4 + 8 + 16 + 32
+ 64 +128)
The following example message is written in the Windows Event Viewer Application
log:
16 4 10797 1 cacao bush nbpem backup of client bush exited with status 71
NetBackup These messages have status codes in the 500 range. Messages
Administration with status codes 500, 501, 502, 503 and 504 begin with "Unable
Console: application to login, status:". Messages with status codes 511 and 512
server status codes may or may not begin with "Unable to login, status:".
and messages
Note: A status code does not always accompany the error
message.
Java exceptions Either the Java APIs or NetBackup Administration APIs generate
these exceptions. These messages begin with the name of the
exception. For example:
java.lang.ClassCastException
or
vrts.nbu.NBUCommandExecutionException
Operating system Any messages that do not match those in the NetBackup
errors documentation are most likely messages from the operating system.
Using logs 59
Troubleshooting error messages in the NetBackup Administration Console
About extra disk space required for logs and temporary files
For successful operation, the NetBackup Administration Console requires extra
disk space to store logs and temporary files. The disk space should be available in
the following locations.
■ On the host that is specified in the logon dialog box
■ In /usr/openv/netbackup/logs/user_ops
■ On the host where the console was started
■ In /usr/openv/netbackup/logs/user_ops/nbjlogs
If space is not available in the respective file systems, you can experience the
following:
■ Long waits for application response
■ Incomplete data
■ No response during logon
■ Reduced functionality in the NetBackup interface, for example, only the Backup,
Archive, and Restore and Files System Analyzer nodes appear in the tree
■ Unexpected error messages:
■ "Cannot connect" socket errors during logon to the NBJava application server
■ "Unable to log in, status: 35 cannot make required directory"
■ "/bin/sh: null: not found (1) "
■ "An exception occurred: vrts.nbu.admin.bpmgmt.CommandOutputException:
Invalid or unexpected class configuration data: <the rest of the message will
vary>"
■ Empty warning dialog boxes
instance of the user service. Additional user service processes are initiated to
process requests from the console.
On both UNIX and Windows, the authentication service is the bpjava-msvc
application. The user service is the bpjava-susvc or bpjava-usvc application. To
enable detailed debug logging, you must first create logging directories for these
applications.
Step 1 Create logging directories On the NetBackup client or server that is specified in the logon dialog
box, create the following directories:
■ bpjava-msvc
■ bpjava-susvc (if a NetBackup server)
■ bpjava-usvc (if a NetBackup client)
Create the directories in the following locations:
■ install_path\NetBackup\logs (Windows)
■ /usr/openv/netbackup/logs (UNIX)
Step 2 Edit the Debug.properties Add the following line to the Debug.properties file:
file
debugMask=0x00040000
■ /usr/openv/java
Change the file on the UNIX machine where you run the jnbSA or
jbpSA commands. The log file name is displayed in the xterm window
where you ran the jnbSA or jbpSA commands.
■ install_path\VERITAS\java
Change the file at this location if you use the NetBackup Windows
Display Console.
Step 3 Edit the nbjava.bat file Perform this step if you use the Windows Display Console on a host
where NetBackup is not installed.
This detailed debug logging provides more information than the NetBackup
Administration Console logging that you can configure within the Administration
Console. See the NetBackup Administrator's Guide, Volume I at the following URL:
http://www.veritas.com/docs/DOC5332
For information on how to create logs when you start the Java-based NetBackup
Administration Console from a Windows computer in NetBackup:
See “About the Java-based administration console logging” on page 163.
Chapter 2
Backup process and
logging
This chapter includes the following topics:
■ Backup process
Backup process
Understanding how the backup process works is a helpful first step in deciding
which processes to review for troubleshooting purposes.
Figure 2-1 illustrates the backup procedure and the process flow during a scheduled
backup.
Backup process and logging 63
Backup process
(3) bprd
NetBackup Jobs
(2) bpdbm nbproxy (1) nbpem (5) bpjobd
Database Database
(8) EMM
(9) bpcd (4) nbjm (6) nbrb (7) nbemm
bpcompatd Database
Active Client
(10) bpbrm (11) bpcd (12) bpbkar Outbound
Data
Connection/
Communication
Paths
Tape only
PBX
(15)txxd/
(13) bptm (14) ltid Tape or Disk
(16)txxcd vnetd
3 The nbjm service communicates with (5) bpjobd, and the job is added to the
job list in the jobs database. The job is now visible in the Activity Monitor, in a
queued state.
4 Once the job has been added to the jobs database, nbjm checks for resources
through the (6) NetBackup Resource Broker (nbrb).
5 The nbrb process secures the required resources from the (7) Enterprise Media
Manager (nbemm) and notifies nbjm that resources have been allocated.
6 After resource allocation, nbjm makes a call to the images database to create
the image files in a temporary location. The required entries in the backup
header tables are also created at this time. The job is now seen as “Active” in
the Activity Monitor.
7 Once the job is active, nbjm uses (8) bpcompatd to open a connection to the
(9) client service (bpcd) on the media server. The bpcompatd service creates
the connection through Private Branch Exchange (PBX) and the NetBackup
Legacy Network Service (vnetd).
8 The bpcd service starts the (10) NetBackup backup and restore manager
(bpbrm).
9 The bpbrm service communicates with (11) bpcd on the client server (through
PBX and vnetd) to start the (12) backup and archive manager (bpbkar). The
bpbrm service also starts the (13) tape management process (bptm).
10 In the case of a tape backup, bptm reserves the drives and issues a mount
request to the (14) logical tape interface daemon (ltid). The ltid service calls
on the (15) robotic drive daemon (txxd, where xx varies based on the type of
robot being used). The txxd daemon communicates the mount request to the
(16) robotic control daemon (txxcd), which mounts the media.
In the case of a disk backup, bptm communicates directly with the disk.
11 The bpbkar service sends the backup data through bptm to be written to the
media storage or the disk storage.
12 When the backup is completed, nbjm is notified and sends a message to
bpjobd. The job now appears as “Done” in the Activity Monitor. The nbjm
service also reports the job exit status to nbpem, which recalculates the next
due time of the job.
Each of the processes that is involved in a backup has an accompanying log file.
These logs can be consulted to diagnose any issues that you encounter with your
backups.
Backup process and logging 65
NetBackup process descriptions
Some additional logs that are not included in the backup process flow but that can
be of use in resolving backup problems include: bpbackup, reqlib, daemon, robots,
and acsssi.
A daemon that executes on all servers and clients is the NetBackup client daemon,
bpcd. On UNIX clients, inetd starts bpcd automatically so no special actions are
required. On Windows clients, bpinetd performs the same functions as inetd.
Note: All NetBackup processes on UNIX can be started manually by running the
following: /usr/openv/netbackup/bin/bp.start_all
Backup process and logging 66
NetBackup process descriptions
The NetBackup master server and the Enterprise media manager (EMM) server
must reside on the same physical host.
The master server is responsible for running jobs as configured in NetBackup
policies by using the services nbpem and nbjm.
The EMM services allocate resources for the master server. The EMM services are
the repository for all device configuration information. The EMM services include
nbemm and its subcomponents along with the nbrb service for device and resource
allocation.
Backup process and logging 67
About backup logging
■ bpbackup
■ daemon
■ reqlib
■ robots
Note: Veritas recommends that the diagnostic level for unified logging be set at the
default level of 6.
See “About global logging levels” on page 50.
Problems with the queued backup jobs that ■ The nbpem log at debug level 3
do not go active ■ The nbjm log at debug level 5
■ The nbrb log at debug level 4
■ The nbproxy log at verbose 4
■ The bpdbm log at verbose 2
■ The nbemm logs at the default levels
■ The mds log at debug level 2
Note: The mds log writes to the nbemm
log.
Backup process and logging 70
Sending backup logs to Veritas Technical Support
Problems with the active backup jobs that do ■ The nbjm log at debug level 5
not write ■ The nbrb log at debug level 4
■ The bpdbm log at verbose 2
■ The bpbrm log at verbose 5
■ The bptm log at verbose 5
■ The bpcd log at verbose 5
If the problem is a tape load or unload issue,
Support may also need the following logs:
■ The ltid log
■ The reqlib log
■ The daemon log
■ The robots log
■ The acsssi log (UNIX only)
See “Setting Media Manager debug logging to a higher level” on page 53.
See “About backup logging” on page 67.
Chapter 3
Media and device
processes and logging
This chapter includes the following topics:
■ Barcode operations
You must know the hosts that are involved to start all the daemons for a robot.
Media and device processes and logging 73
Media and device management process
Automated
ltid acsd acsssi Cartridge System
acsel
Optical Disk
odld Library
Tape Library
vmd avrd tl4d 4mm
Tape Library
tl8d tl8cd 8mm
Tape Library
tldd tldcd DLT
Tape Library
tlhd tlhcd Half-inch
Tape Stacker
tshd Half-inch
Note: When you mount a tape on UNIX, the drive_mount_notify script is called.
This script is in the /usr/openv/volmgr/bin directory. Information on the script can
be found within the script itself. A similar script is called for the unmount process
(drive_unmount_notify, in the same directory).
When a robotic volume is added or removed through the media access port, the
media management utility communicates with the appropriate robotic daemon to
verify the volume location or barcode. The media management utility (through a
library or command-line interface) also calls the robotic daemon for robot inventory
operations.
Figure 3-2 shows an example of the media and device management process.
Media and device processes and logging 75
Shared Storage Option management process
EMM
Database Device-
bptm management
utility
Request
tape mount
Media-
ltid nbemm management
utility
Mo
un Inventory
tm
ed barcodes
ia
ID or inject/eject
Robotic
Non-robotic drives control
Tape library
Enterprise Media Manager Media TL8
(Master Server) Server
■ nbrb tells the device allocator (DA) in the EMM server to stop scanning the
selected drive.
■ nbemm tells the appropriate media server (the scan host for the selected drive)
to stop scanning the drive. The stop scan request is carried out by means of
oprd, ltid, and avrd in the media server’s shared memory.
Media and device processes and logging 76
Shared Storage Option management process
■ nbemm informs nbrb when the scanning on the selected drive has stopped.
■ nbrb informs nbjm that the selected drive (A) is available for the backup.
■ nbjm conveys the mount request and drive selection to bptm, which proceeds
with the backup. To protect the integrity of the write operation, bptm uses SCSI
reservations.
For more information about how NetBackup reserves drives, see the NetBackup
Administrator’s Guide, Volume II.
■ The mount-media operation is initiated.
■ bptm makes position checks on the drive to ensure that another application has
not rewound the drive. bptm also does the actual write to the tape.
■ When the backup is complete, nbjm tells nbrb to release resources.
■ nbrb de-allocates the drive in EMM.
■ EMM tells the scan host to resume scanning the drive. The scan request is
carried out by means of oprd, ltid, and avrd in the media server’s shared
memory.
Figure 3-3 illustrates the Shared Storage Option management process.
Media and device processes and logging 77
Barcode operations
Figure 3-3 Media and device management process flow showing SSO
components
EMM
Backup and Database
Device
archive
monitor
process
Device-
nbjm nbrb nbemm/DA management
utility
an
sc
p
Sto
bptm
Barcode operations
Barcode reading is mainly a function of the robot hardware instead of media and
device management. When a robot has a barcode reader, it scans any barcode
that is on a tape and stores the code in its internal memory. This associates the
slot number and the barcode of the tape in that slot. NetBackup determines that
association for its own use by interrogating the robot.
If a robot supports barcodes, NetBackup automatically compares a tape’s barcode
to what is in the EMM database as an extra measure of verification before you
Media and device processes and logging 78
Barcode operations
mount the tape. A request for the media that is in a robot that can read barcodes
begins in the same manner as other requests.
See Figure 3-4 on page 79.
The ltid command includes the media ID and location information in a mount
request to the robotic daemon for the robot that has the media ID. This request
causes the robotic daemon to query the robotic-control daemon or the robot for the
barcode of the tape in the designated slot. (This is a preliminary check to see if the
correct media is in the slot.) The robot returns the barcode value it has in memory.
The robotic daemon compares this barcode with the value it received from ltid
and takes one of the following actions:
■ If the barcodes don’t match, and the mount request is not for a NetBackup
backup job, the robotic daemon informs ltid and a pending action request
(Misplaced Tape) appears in the Device Monitor. An operator must then insert
the correct tape in the slot.
■ If the barcodes don’t match and the mount request is for a NetBackup backup
job, the robotic daemon informs ltid and the mount request is canceled.
NetBackup (bptm) then requests a new volume from nbjm and from EMM.
■ If the barcodes match, the robotic daemon requests the robot to move the tape
to a drive. The robot then mounts the tape. At the start of the operation, the
application (for example, NetBackup) checks the media ID and if it also matches
what should be in this slot, the operation proceeds. For NetBackup, a wrong
media ID results in a “media manager found wrong tape in drive” error
(NetBackup status code 93).
Media and device processes and logging 79
Media and device management components
EMM
Request Media ID Database
mount
ltid nbemm
Media-
Mount media vmd management
ID utility
Robot inventory
request or inject
tl8d tl8cd
1 2 3
What is Mount
Barcode
barcode tape
NetBackup server
/usr/openv/volmgr/
/usr/openv/volmgr/bin/ /usr/openv/volmgr/debug/1
Table 3-1 describes the directories and files that are of special interest.
Table 3-1 Media and device management directories and files (continued)
Table 3-2 describes the media management and device management programs
and daemons. The explanations include what starts and stops the program or
daemon, and the log (if any) where it records its activities. On UNIX, all of the
components that are discussed in this table are in /usr/openv/volmgr/bin. On
Windows, they are in install_path\volmgr\bin.
Note: The following table contains references to the system log. On UNIX, syslog
manages this log (the facility is daemon). On Windows, the Event Viewer manages
the system log (the log type is Application).
Program or Description
daemon
Debug Log: Errors are logged in the system log and robots debug
log. Debug information is included by adding VERBOSE to the
vm.conf file. On UNIX, debug information is also included by
starting the daemon with the -v option: this option can also be used
through ltid, or by putting VERBOSE in the vm.conf file.
Program or Description
daemon
Debug Log: All errors are logged in the system log. Debug
information is included by adding VERBOSE to the vm.conf file.
On UNIX, debug information is also included by aborting avrd and
starting the daemon with the -v option.
Program or Description
daemon
tl4d The Tape Library 4MM daemon is the interface between ltid and
the Tape Library 4MM and communicates with the robotics through
a SCSI interface.
Started By: Starting ltid (or on UNIX, independently by using
the /usr/openv/volmgr/bin/tl4d command).
Debug Log: All errors are logged in the system log. Debug
information is included by adding VERBOSE to the vm.conf file.
On UNIX, debug information is also included by starting the daemon
with the -v option (either by itself or through ltid).
tl8d The Tape Library 8MM daemon provides the robotic control for a
TL8 robot (Tape Library 8mm or Tape Stacker 8mm). The Tape
Library 8MM daemon drives in the same TL8 robot can be attached
to different hosts than the robotic control. tl8d is the interface
between the local ltid and the robotic control. If a host has a
device path for a drive in a TL8 robot, then mount or unmount
requests for that drive go first to the local ltid and then to the local
tl8d (all on the same host). tl8d then forwards the request to tl8cd
on the host that is controls the robot (it can be on another host).
Debug Log: Errors are logged in the system log and the robots
debug log. Debug information is included by adding VERBOSE to
the vm.conf file. On UNIX, debug information is also included by
starting the daemon with the -v option (either by itself or through
ltid).
Media and device processes and logging 84
Media and device management components
Program or Description
daemon
tl8cd The Tape Library 8MM control daemon provides the robotic control
for a TL8 robot and communicates with the robotics through a SCSI
interface. tl8cd receives mount and unmount requests from tl8d on
the host to which the drive is attached and then communicates
these requests to the robot.
Started By: Starting ltid (or on UNIX, independently by using
the /usr/openv/volmgr/bin/tl8cd command).
Debug Log: Errors are logged in the system log and the robots
debug log. Debug information is included by adding VERBOSE to
the vm.conf file. On UNIX, debug information is also included by
starting the daemon with the -v option (either by itself or through
ltid).
tldd The Tape Library DLT daemon works with tldcd to handle requests
to TLD robots (Tape Library DLT and Tape Stacker DLT). tldd
provides the interface between the local ltid and the robotic
control (tldcd) in the same way as explained previously for tl8d.
Started By: Starting ltid (or on UNIX, independently by using
the /usr/openv/volmgr/bin/tldd command).
Debug Log: Errors are logged in the system log and robots debug
log. Debug information is included by adding VERBOSE to the
vm.conf file. On UNIX, debug information is also included by
starting the daemon with the -v option (either by itself or through
ltid).
Media and device processes and logging 85
Media and device management components
Program or Description
daemon
tldcd The tape library DLT control daemon provides robotic control for a
TLD robot in the same way as explained previously for tl8cd.
Started By: Starting ltid (or on UNIX, independently by using
the /usr/openv/volmgr/bin/tldcd command).
Debug Log: Errors are logged in the system log and robots debug
log. Debug information is included by adding VERBOSE to the
vm.conf file. On UNIX, debug information is also included by
starting the daemon with the -v option (either by itself or through
ltid).
tlhd The Tape Library Half-inch daemon works with tlhcd to handle
requests to the TLH robots that are in an IBM Automated Tape
Library (ATL). tlhd provides the interface between the local ltid and
the robotic control (tlhcd) in the same way as explained previously
for tl8d.
Started By: Starting ltid (or on UNIX, independently by using
the /usr/openv/volmgr/bin/tlhd command).
Debug Log: Errors are logged in the system log and robots debug
log. Debug information is included by adding VERBOSE to the
vm.conf file. On UNIX, debug information is also included by
starting the daemon with the -v option (either by itself or through
ltid).
tlhcd The Tape Library half-inch control daemon provides robotic control
for a TLH robot that is in an IBM Automated Tape Library (ATL) in
the same way as explained previously for tl8cd.
Debug Log: Errors are logged in the system log and robots debug
log. Debug information is included if the daemon is started with the
-v option (either by itself or through ltid). The -v option is
available only on UNIX. Also, add the VERBOSE option to the
vm.conf file.
Media and device processes and logging 86
Media and device management components
Program or Description
daemon
tlmd The Tape Library Multimedia daemon is the interface between ltid
and a TLM robot that is in an ADIC Distributed AML Server (DAS).
This daemon communicates with the TLM robotics through a
network API interface.
Debug Log: Errors are logged in the system log and robots debug
log. Debug information is included if the daemon is started with the
-v option (either by itself or through ltid). The -v option is
available only on UNIX. Also, add the VERBOSE option to the
vm.conf file.
tshd The Tape Stacker Half-inch daemon is the interface between ltid
and the half-inch-cartridge stacker and communicates with the
robotics through a SCSI interface. This robot is not supported on
Windows.
Started By: Starting ltid (or on UNIX, independently by using
the /usr/openv/volmgr/bin/tshd command).
Debug Log: System log and also a debug log if the daemon or
reqlib debug directories exist.
Media and device processes and logging 87
Media and device management components
Program or Description
daemon
vmscd The Media Manager Status Collector Daemon keeps the EMM
server database up-to-date with the actual status of the drives that
are attached to the 5.x servers.
■ Restore process
Restore process
Understanding how the restore process works is a helpful first step in deciding
which logs to gather for a particular issue. The restore process differs depending
on whether you restore an image from tape or from disk.
Figure 4-1 illustrates a restore from tape.
Restore process and logging 89
Restore process
(2) bprestore
NetBackup
(3) bpdbm (1) bprd (4) bpjobd Jobs Database
Database
Outbound
Connection/
Communication
Paths
(7) tar (11) ltid (9) nbrb
PBX
Some additional logs that are not included in the restore process flows but that can
be of use in resolving restore problems include: reqlib, daemon, robots, and
acsssi.
(2) bprestore
NetBackup
(3) bpdbm (1) bprd (6) bpjobd Jobs Database
Database
(9) bptm (7) bpbrm (4) bprd child (5) nbemm EMM Database
Active Client
(8) tar
Data PBX vnetd
3 The bprd process initiates a (4) child bprd process. The child bprd process
makes a call to the (5) Enterprise Media Manager (nbemm) to verify that the
disk storage unit is available.
4 The child bprd process communicates with (6) bpjobd to allocate a jobid.
The restore job is now visible in the Activity Monitor.
5 The bprd process starts the (7) NetBackup backup and restore manager (bpbrm)
on the media server, through Private Branch Exchange (PBX) and the NetBackup
Legacy Network Service (vnetd).
6 The bpbrm service uses PBX and vnetd to establish a connection with the (8)
Tape Archive program (tar) on the client system. It also starts the (9) tape
management process (bptm).
7 The bptm process makes a call to bpdbm (through PBX and vnetd) to get the
fragment information and then mounts the disk.
8 The bptm process reads the backup image from the disk and streams the
requested data to tar.
9 The tar process commits the data to the storage destination.
Each of the processes that is involved in a restore has an accompanying log file.
These logs can be consulted to diagnose any issues that you encounter with your
restore.
Query Query
Master Client
Server Server
The following are the processing steps in a restore (in the order presented):
■ When the user starts a restore, NetBackup invokes the client’s bprestore
program which sends a request to the request daemon, bprd. This request
identifies the files and client. The request daemon then uses bpcd (client daemon)
to start the backup and restore manager (bpbrm).
Note: To restore Backup Exec images, bpbrm initiates mtfrd instead of nbtar
on the clients. The server processes are the same as those used for NetBackup
restores.
■ If the disk device or tape device on which the data resides attaches to the master
server, the following occurs: bprd starts the backup and restore manager on
the master server. If the disk unit or tape unit connects to a media server, bprd
starts the backup and restore manager on the media server.
■ The backup and restore manager starts bptm and uses the client daemon (bpcd)
to establish a connection between NetBackup nbtar on the client and bptm on
the server.
■ For tape: The bptm process identifies which media is needed for the restore,
based on the image catalog. bptm then requests the allocation of the required
media from nbrb through nbjm. nbjm then asks mds (part of nbemm)for the
Restore process and logging 94
Windows client restore
resources. nbemm allocates the media and selects and allocates an appropriate
drive (for tape media).
bptm asks ltid to mount the tape in the drive.
For disk: bptm does not need to ask nbrb for an allocation, because disk
inherently supports concurrent access. bptm uses the file path in a read request
to the system disk manager.
■ bptm directs the image to the client in one of two ways. If the server restores
itself (server and client are on the same host), nbtar reads the data directly
from shared memory. If the server restores a client that resides on a different
host, it creates a child bptm process which transmits the data to nbtar on the
client.
Note: Only the part of the image that is required to satisfy the restore request
is sent to the client, not necessarily the entire backup image.
■ The NetBackup nbtar program writes the data on the client disk.
Note: PBX must be running for NetBackup to operate (PBX is not shown in the
next diagram). See the NetBackup Troubleshooting Guide for more information on
how to resolve PBX problems.
NetBackup
nbwin bprd
user interface
bpinetd bpbrm
bpcd bptm
tar32
Backup Image
Client Disk
Note: Veritas recommends that the diagnostic level for unified logging be set at the
default level of 6.
See “About global logging levels” on page 50.
Restore process and logging 97
Sending restore logs to Veritas Technical Support
Problems with restore jobs from tape ■ The nbjm log at debug level 5
■ The nbemm log at debug level 1
■ The nbrb log at debug level 4
■ The bpdbm log at verbose 1
■ The bprd log at verbose 5
■ The bpbrm log at verbose 5
■ The tar log at verbose 5
■ The bpcd log at verbose 5
If the problem is a media or a drive issue,
Support may also need the following logs:
■ The reqlib log
■ The daemon log
■ The robots log
■ The acsssi log (UNIX only)
Problems with restore jobs from disk ■ The bpdbm log at verbose 1
■ The bprd log at verbose 5
■ The bpbrm log at verbose 5
■ The bptm log at verbose 5
■ The bpdm log at verbose 5
■ The tar log at verbose 5
■ The bpcd log at verbose 5
See “Setting Media Manager debug logging to a higher level” on page 53.
See “About restore logging” on page 95.
Chapter 5
Advanced Backup and
Restore Features
This chapter includes the following topics:
■ Synthetic backups
(1) NetBackup
bpbackup
(2) bpdbm (2) nbpem (1) bprd User Interface or
(script)
command line
(4) bpcd
Shared
memory
Shared (7,8)
Disk STU (7,10) nbftclnt EMM Database
memory nbftsrvr
Control Path
Master Media Client
Server Server Server
Data Path
The processing steps for a SAN client backup operation are the following:
SAN client backup procedure
1 The NetBackup master server or primary client initiates the backup. The
NetBackup request daemon (bprd) submits a backup request to the Policy
Execution Manager (nbpem). nbpem processes the policy configurations.
All other daemons and programs are started as necessary including nbpem,
nbjm, nbrb, and nbemm.
3 The Job Manager service (nbjm) requests backup resources from the Resource
Broker (nbrb), that returns information on the use of shared memory for the
SAN client.
4 The nbjm service starts the backup by means of the client daemon bpcd, which
starts the backup and restore manager bpbrm.
5 The bpbrm service starts bptm, which does the following:
■ Requests the SAN client information from nbjm.
■ Sends a backup request to the FT server process (nbftsrvr).
■ Sends a backup request to the FT client process on the client (nbftclnt),
that does the following: Opens a Fibre Channel connection to nbftsrvr on
the media server, allocates the shared memory, and writes the shared
memory information to the backup ID file.
6 The bpbrm service uses bpcd to start bpbkar, that does the following:
■ Reads the shared memory information from the BID file (waits for the file
to exist and become valid).
■ Sends the information about files in the image to bpbrm.
■ Writes the file data to bpbkar, optionally compresses it, then writes the data
to the shared buffer.
■ Sets the buffer flag when the buffer is full or the job is done.
7 The FT client process (nbftclnt) waits for the shared memory buffer flag to
be set. It then transfers the image data to the FT Server (nbftsrvr) shared
memory buffer, and clears the buffer flag.
8 The nbftsrvr service waits for data from nbftclnt; and writes the data is
written to the shared memory buffer. When the transfer completes, nbftsrvr
sets the buffer flag.
9 bptm waits for the shared memory buffer flag to be set, writes data from the
buffer to the storage device, and clears the buffer flag.
10 At the end of the job:
■ bpbkar informs bpbrm and bptm that the job is complete.
Advanced Backup and Restore Features 101
SAN Client Fiber Transport restore
■ nbftclnt closes the Fibre Channel connection and deletes the BID file.
NetBackup
nbjm bprd bprestore user interface
Command
bpcd
line
Backup Image
(Tape or Disk)
Backup Image
UNIX: NetBackup
Shared Shared
bptm child nbtar
Memory Memory Windows: tar32
The process flow for a SAN client restore is as follows (in the order presented).
■ When the user starts a restore, NetBackup invokes the client’s bprestore
program that sends a request to the request daemon, bprd. This request
identifies the files and client. The request daemon then uses bpcd (client daemon)
to start the backup and restore manager (bpbrm).
■ If the disk or tape where the data resides attaches to the master server, then
bprd starts the backup and restore manager on the master server. If the disk
unit or tape unit connects to a media server, bprd starts the backup and restore
manager on the media server.
■ bpbrm starts bptm and provides bptm with the backup ID and the shmfat (shared
memory) flag.
■ bptm does the following:
■ Requests the SAN client information from the Job Manager service (nbjm).
■ Sends a restore request to the FT server process (nbftsrvr).
■ Sends a restore request to the FT client process on the client (nbftclnt).
nbftclnt opens a Fibre Channel connection to nbftsrvr on the media
server, allocates the shared memory, and writes the shared memory
information to the backup ID file.
■ bpbrm starts tar by means of bpcd and provides tar with the backup ID, socket
information, and the shmfat (shared memory) flag.
■ bptm does the following:
■ The FT client (nbftclnt) waits for the data from nbftsrvr and writes the data
to the shared memory buffer on the client. nbftclnt then sets the buffer flag.
■ At the end of the job:
■ bptm informs tar and bpbrm that the job is complete.
■ nbftclnt closes the Fibre Channel connection and deletes the BID file.
bprd bpbackupdb
nbpem
nbjm
Back up relational database files
3 Back up NetBackup database files
/usr/openv/db/ bprd
staging
Master Server
NetBackup creates the disaster recovery file, and emails it to the administrator
if the email option was selected in the policy.
Consult the following logs for messages on hot catalog backup:
■ bpdbm, bpbkar, bpbrm, bpcd, bpbackup, bprd
For messages pertaining only to the relational database files, see the EMM server.log
file and the bpdbm log file in the following directories:
■ UNIX: /usr/openv/netbackup/logs/bpdbm
/usr/openv/db/log/server.log
■ Windows: install_path\NetBackup\logs\bpdbm
install_path\NetBackupDB\log\server.log
NetBackup Command
recovery wizard line
bprd bprecover
2 3 4
See “Restore
from tape (UNIX)”
or “Restore from Restore
disk”, depending Relational
on the catalog /usr/openv/db/ Relational
Database Files
backup policy staging database files
See “Restore
from tape (UNIX)”
or “Restore from
disk”, depending
on the catalog Master Server
backup policy
A restore of the NetBackup database and relational database (NBDB) files from a
hot catalog backup consists of the following steps (in the order presented):
■ The NetBackup catalog image and configuration files are restored.
■ The NBDB files are restored. The database files are restored to
/usr/openv/db/staging (UNIX), or to install_path\NetBackupDB\staging
(Windows).
■ After the files are restored to the staging directory, NBDB is recovered.
■ The NBDB files are moved from the staging directory to a location that is
determined by the following: The bp.conf file VXDBMS_NB_DATA setting on UNIX
and by the corresponding registry key on Windows. The default location is
/usr/openv/db/data on UNIX, and install_path\NetBackupDB\data on
Windows.
If the relational database files are relocated, they are moved from the staging
directory to the /usr/openv/db/data/vxdbms.conf file (UNIX) or the
install_path\NetBackupDB\data\vxdbms.conf file (Windows). For information
Advanced Backup and Restore Features 107
Synthetic backups
on how to relocate the NetBackup relational database files after installation, see
the NetBackup Administrator’s Guide, Volume I.
Synthetic backups
The typical NetBackup backup process accesses the client to create a backup. A
synthetic backup is a backup image created without using the client. Instead, a
synthetic backup process creates a full or a cumulative incremental image by using
previously created backup images called component images.
For example, an existing full image and subsequent differential incremental images
can be synthesized to create a new full image. The previous full image and the
incrementals are the component images. The new synthetic full image behaves
like a backup that is created through the traditional process. The new synthetic full
image is a backup of the client that is as current as the last incremental. The
synthetic image is created by copying the most current version of each file from the
most recent component image that contains the file. A synthetic backup must be
created in a policy with the True Image Restore with Move Detection option
selected. This option enables the synthetic backup to exclude the files that have
been deleted from the client file system from appearing in the synthetic backup.
Like a traditional backup, nbpem initiates a synthetic backup. It submits a request
to nbjm to start the synthetic backup process and nbjm then starts bpsynth, which
executes on the master server. It controls the creation of the synthetic backup image
and the reading of the files that are needed from the component images. If directory
bpsynth exists in the debug log directory, additional debug log messages are written
to a log file in that directory.
bpsynth makes a synthetic image in several phases:
Advanced Backup and Restore Features 108
Synthetic backups
Table 5-1
Phase Description
nbpem
nbjm
Request to make
Synthetic backup
2 - Obtain In phase 2, bpsynth obtains write resources (storage unit, drive, and media)
resources for the new image. It also reserves all the read media containing component
images and obtains the drive for the first media to be read.
When the component images reside on BasicDisk, no resource reservation
is done.
Advanced Backup and Restore Features 109
Synthetic backups
Phase Description
3 - Copy In phase 3, bpsynth starts the writer bptm (for tape and disk) on the media
data server to write the new synthetic image. It also starts a reader bptm (tape)
or bpdm (disk) process for each component image on a media server that
can access the component image. The reader process reads all extents for
the component image.
child
bptm
Data flow
child bptm
or bpdm
parent bptm Component
or bpdm image(s)
Master Media
Server Server
Note that bpsynth only starts the parent bptm (writer) and bpdm (reader)
process on the media server. The parent in turn starts a child process. The
parent and child communicate by means of buffers in shared memory.
The bpsynth process sends the extents (starting block and count) for each
component image to the corresponding child bptm or bpdm reader process.
The parent bptm or bpdm reader process reads the data from the appropriate
media into the shared buffers. The child bptm or bpdm reader process sends
the data in the shared buffers to the child bptm writer process over a socket.
The child bptm writer process writes the data into the shared buffers. The
parent bptm writer process copies the data from the shared buffers to the
media and notifies bpsynth when the synthetic image is complete.
4 - Validate In phase 4, the bpsynth process validates the image. The new image is now
the image visible to NetBackup and can be used like any other full or cumulative
incremental backup.
Synthetic backup requires that true image restore (TIR) with move detection
be selected for each component image, and that the component images are
synthetic images.
Advanced Backup and Restore Features 110
Synthetic backups
Step 3 Change the Global In Host Properties, select a master server and set the Global logging level to
logging level. 5.
See the NetBackup Troubleshooting Guide for more information on how to use the
Host Properties window to access configuration settings.
Step 4 Rerun the job. Rerun the job and gather the logs from the directories that you created.
The bptm logs are required only if the images are read from or written to a tape
device or disk. The bpdm logs are needed only if the images are read from disk.
If the images are read from multiple media servers, the debug logs for bptm or
bpdm must be collected from each media server.
See “Logs to accompany problem reports for synthetic backups” on page 110.
install_path/netbackup/db/jobs/trylogs/jobid.t
If the job ID of the synthetic backup job was 110, the try file is named
110.t.
Policy attributes Use the following command to capture the policy attributes:
install_path/netbackup/bin/admincmd/bppllist
policy_name -L
where policy_name is the name of the policy for which the synthetic
backup job was run.
List of storage Capture the list of storage units from the following command:
units
install_path/netbackup/bin/admincmd/bpstulist -L
See “Creating legacy log directories to accompany problem reports for synthetic
backup” on page 110.
Chapter 6
Storage logging
This chapter includes the following topics:
nbjm bpbackup
nbpem (2) bprd (2)
(2,3,5) (1)
Image Jobs DB
nbemm
nbrb (3,4) EMM DB
Catalog
(3,4) (MDS)
bpdbm bpjobd
NDMP Server
bpbrm ndmpagent
bpcd (5) Application Local Disk
(5,6) (7) (8-Local)
NDMP Server
ltid (6,7) bptm (6) Application
NDMP Host
The basic processing steps for an NDMP backup operation are the following:
NDMP backup procedure
1 The NetBackup administrator runs the bpbackup command to start the backup
job. Or, a scheduled policy that is created on the NetBackup Administration
Console can initiate the job.
2 The bpbackup process connects to the master server and creates the backup
request. The NetBackup Request Manager (bprd) sends the backup request
to the Policy Execution Manager (nbpem), who submits the job to the Job
Manager (nbjm).
3 nbjm requests resources from the Resource Broker (nbrb) that are required
to run the job. nbrb accesses the Media and Device Selection (MDS) of the
Enterprise Media Management (nbemm) to evaluate the resources request.
MDS queries the EMM database to identify the resources to use for this job.
4 MDS provides nbrb with a list of resources for the job, and nbrb passes it on
to nbjm.
5 nbjm initiates communication with the media server that is associated with this
backup job. It goes through the client service (bpcd) to start the Backup and
Restore Manager (bpbrm) on the media server.
6 bpbrm starts the Tape Manager (bptm) on the media server. Eventually, the
parent bptm process makes a request to ltid to mount the tape to be used
for the backup job.
7 On the NetBackup for NDMP server, one of the following occurs: sends the
necessary NDMP SCSI robotic commands to mount the requested tape on the
storage device.
■ The NDMP agent service (ndmpagent) connects to the filer that issues the
NDMP commands to mount the tape that is directly attached.
■ ltid on the media server issues the necessary NDMP SCSI robotic
commands to mount the requested tape on the storage device.
that houses the data to be backed up to the NDMP server that writes the
backup to its tape storage.
■ Remote backup (not shown in the process flow diagram). The device that
is used to write the backup is associated with a NetBackup storage unit.
bptm on the NetBackup media server mounts a tape on a tape drive.
NetBackup sends the NDMP commands to the NDMP server to initiate the
backup to the non-NDMP media manager storage unit. The data travels
over the network from the NDMP host to the NetBackup media server,
which writes the data to the selected storage unit.
9 Throughout the backup operation and at its completion, the NDMP server sends
status about the backup operation to the NetBackup for NDMP server. Several
NetBackup processes send information about the job to bpjobd, that uses this
information to update the job status that you can view in the NetBackup Activity
Monitor.
Status, catalog, and other job information movement is shown in dashed lines
in the process flow diagram.
Backup, Archive,
nbjm (2,3) nbpem (1) bprd (1)
and Restore (1)
bpbackup (1)
local restore
bpbrm ndmpagent
bpcd (3)
(3,4,6) (4,5,6)
NDMP host 1
bpjobd (6)
Remote restore 3-way restore
over the network over the network
NetBackup
Master
for NDMP NDMP Host Status Info
Server
Server
The basic processing steps for an NDMP restore operation are as follows:
NDMP restore procedure
1 An administrator at the NetBackup Administration Console on a NetBackup
master server or media server initiates a restore job by browsing the images
catalog and by selecting the files to be restored from the NDMP images. This
process is similar to selecting files to be restored from standard backup images.
The NetBackup master server identifies the specific media that is required to
perform the restore. In this diagram, the media is a tape volume.
2 After the master server identifies the data to be restored and the media required,
it submits a restore job. The Job Manager (nbjm) then requests the required
resources. This resource request causes the allocation of the media that
contains the data to be restored. In this example, a tape drive is used during
the restore operation.
Storage logging 117
NDMP restore logging
3 The master server contacts the media server that participates in the restore
job, and starts the Restore Manager (bpbrm) process to manage the restore
job. bpbrm starts the Tape Manager process (bptm), that queries nbjm for the
tape volume. Then, bptm requests that the logical tape interface daemon (ltid)
mounts the tape.
4 On the NetBackup for NDMP server, the NDMP agent (ndmpagent) connects
to the filer and issues NDMP commands to mount the tape that is directly
attached, and ltid sends NDMP commands to mount the requested tape on
the storage device. Or, the media server itself issues tape mount requests
much like a regular media manager storage unit.
5 One of the following occurs, depending on the type of NDMP restore operation:
■ Local restore. NetBackup sends the NDMP commands to the NDMP server
to initiate the restore operation from a tape drive to a local disk. The restore
data travels from a tape drive to a local disk on the NDMP host without
traversing the LAN.
■ Three-way restore. The NetBackup media server establishes NDMP
communications with both of the NDMP servers that are involved in the
restore. To initiate the restore of data from tape on one NDMP server to
disk storage on the other NDMP server, the media server sends NDMP
commands to both NDMP servers. The restore data travels over the network
between the NDMP hosts.
■ Remote restore. NetBackup sends the NDMP commands to the NDMP
server to prepare the server for the restore. bptm on the media server reads
the restore data from tape and sends it over the network to the NDMP host
where the data is written to disk storage.
6 The NDMP server sends status information about the restore operation to the
NetBackup for NDMP server. Various NetBackup processes (nbjm, bpbrm,
bptm, and others) send job status information to the master server. The Jobs
Database Manager (bpjobd) process on the master server updates the restore
job status in the jobs database. You can view this status in the Activity Monitor.
Chapter 7
NetBackup Deduplication
logging
This chapter includes the following topics:
■ Logging keywords
is processed into the crdb later when the next content router queue processing
job runs. Beginning with NetBackup 7.6, .tlog files do not contain additions to
the database.
The functional overview is as follows:
Windows
& Java GUI
& CLIs
nbsl
nbsl
tpconfig/vmd
DSM api to manipulate Storage Servers
spad
EMM DB
spoold
In this scenario, the client backs up data directly to the media server and the media
server deduplicates the data before it stores it locally. Ensure that this is on the
correct media server which is not always the same as the MSDP storage server
(due to load balancing).
For deduplication-specific logging, enable the following on the media server:
1. Verbose 5 bptm logging:
■ Create a log directory named bptm in /usr/openv/netbackup/logs
(Windows: install_path\NetBackup\logs)
■ Set the bptm log verbosity to 5 in the NetBackup Administration Console.
To do this, click on Host Properties > Logging for the media server. If you
use UNIX/Linux, set the bptm log verbosity to 5 in the
/usr/openv/netbackup/bp.conf file by appending the following line:
BPTM_VERBOSE = 5
NetBackup Deduplication logging 120
Deduplication backup process to the Media Server Deduplication Pool (MSDP)
LOGLEVEL = 10
Note: You can also modify DEBUGLOG in the pd.conf file to specify a path
to which to log; however, we recommend leaving the DEBUGLOG entry
commented out. The logging information (PDVFS debug logging) then logs
to the bptm and bpdm logs.
■ Ensure that you are on the correct media server and restart the MSDP
storage server services.
5. Extract the bptm PID found in step 3 from the bptm log on the media server.
This step only gathers the single-line entries; review the raw logs to see the
multi-line log entries. In the following examples, 3144 is the bptm PID:
■ Windows command line:
6. Gather the spoold session logs that cover the dates from when the backup
started and when it failed from the following logs:
Windows:
<dedup_path>\log\spoold\<mediasvr_IP_or_hostname>\bptm\Receive\MMDDYY.log
<dedup_path>\log\spoold\<mediasvr_IP_or_hostname>\bptm\Store\MMDDYY.log
UNIX/Linux:
<dedup_path>/log/spoold/<mediasvr_IP_or_hostname>/bptm/Receive/MMDDYY.log
<dedup_path>/log/spoold/<mediasvr_IP_or_hostname>/bptm/Store/MMDDYY.log
install_path\NetBackup\logs\nbostpxy
UNIX/Linux:
/usr/openv/netbackup/logs/nbostpxy
5. tpcommand (...\volmgr\debug\tpcommand)
6. storage_directory\log\msdp-config.log
■ storage_directory\log\pdde-config.log
■ storage_directory\log\pdde-config.log
NetBackup Deduplication logging 123
Media server deduplication/pdplugin logging
printcmds=true
printCmdLines=true
debugMask=0x0C000000
debugOn=true
■ tpcommand (...\volmgr\debug\tpcommand)
■ storage_directory\log\msdp-config.log
For backups:
■ Enable verbose 5 bptm on the media servers to read/write backups
■ Uncomment LOGLEVEL = 10 in the media server pd.conf file
For duplications or replications:
■ Enable verbose 5 bpdm on the media server(s) to read/write duplications
■ Uncomment LOGLEVEL = 10 in the media server pd.conf file
NetBackup Deduplication logging 124
Disk monitoring logging
■ Enable trace level spad and spoold logging so that the failing duplication or
replication job can be traced across bpdm/pdvfs > source spad/spoold session
log > source replication.log > target spad/spoold session logs
Logging keywords
Support uses the following keywords when it reviews the logs.
Keyword Description
get_agent_cfg_file_path_for_mount Uses the PureDisk agent configuration file (note the .cfg
file name); determines short name or FQDN.
Recommended routing table CR routing table for the CR's to route fingerprint/so's; more
useful when PDDO targets PureDisk.
Keyword Description
Windows
& Java GUI
& CLIs
nbsl
nbsl
tpconfig/vmd
DSM API to manipulate storage servers
In this scenario, the client backs up the data directly to the media server and the
media server accesses the vendor plug-in to transfer the data to the storage server.
For logging that is specific to OST, enable the following on the media server or
plug-in host:
1. In the registry or bp.conf file, set VERBOSE = 5.
2. Ensure that the following directories exist under /usr/openv/netbackup/logs
(for Windows, use install_path\NetBackup\logs):
■ bptm
■ bpbrm
■ bpstsinfo
9. Extract the bptm PID found in step 8 from the bptm log on the media server.
This step gathers only the single-line entries; review the raw logs to see the
multi-line log entries. In the following examples, 3144 is the bptm PID:
■ Windows command line:
10. Gather the vendor specific plug-in logs that cover the dates from when the
backup started and when it failed.
the OST partner and it resides on the media server for use by NetBackup. NetBackup
depends on the OST plug-in for a path to the storage server.
Communication to the storage server is through the network; name resolution on
the media server and the storage server must be configured correctly. All supported
vendor plug-ins can communicate over a TCP/IP network and some can also
communicate to the disk storage on a SAN network.
To determine the capabilities of a disk appliance, NetBackup uses the plug-in to
query the storage appliance. The capabilities can include deduplicated storage,
optimized off-host duplication, and synthetic backups.
Each OST vendor can report different log messages. A review of the bptm log and/or
plug-in log for a backup or a restore job is the best way to understand the specific
calls made to the storage server through the plug-in.
The basic steps include the following:
■ Claim the resource
■ sts open_server
■ Create the image
■ write
■ close
■ sts close_server
The example of calls in a vendor plug-in log are as follows:
After the initial configuration of the diskpools, you must run the nbdevconfig
-updatedp command as follows to recognize any new flag that the vendor added:
To manually add the supported flags, you can use the following commands:
■ nbdevconfig -changests -storage_server storage server name -stype
OST_TYPE -setattribute OptimizedImage
You should also review the following flag for the storage server:
■ OptimizedImage - supports accelerator
OpenStorage Technology (OST) logging 131
OpenStorage Technology (OST) configuration and management
To list the OpenStorage credentials for all of the media servers, use the following
commands:
■ UNIX/Linux: /usr/openv/volmgr/bin/tpconfig -dsh -all_hosts
■ Windows: install dir\volmgr\bin\tpconfig -dsh -all_hosts
Chapter 9
Storage lifecycle policy
(SLP) and Auto Image
Replication (A.I.R.) logging
This chapter includes the following topics:
■ About storage lifecycle policies (SLPs) and Auto Image Replication (A.I.R.)
Understanding the storage lifecycle policy (SLP) operations (for example, backup,
duplication, replication, import, and snapshot) can help determine which logs can
be used to troubleshoot an issue. This topic primarily focuses on the Automatic
Image Replication (A.I.R.) and duplication process flows. The process flow for other
operations, like backups and snapshots, are covered in other topics of this guide.
See the NetBackup Administrator’s Guide, Volume I for more information about
SLPs and A.I.R.
Master Server
IsWindowOpen nbsetserv
Lock/Unlock copy
bpduplicate bpdbm
Media Server
Create
fragment
bpdm bptm
Read/Write
offset
Storage lifecycle policy (SLP) and Auto Image Replication (A.I.R.) logging 134
Automatic Image Replication (A.I.R.) process flow logging
4. nbjm starts nbreplicate (nbreplicate appears in the admin log) and passes
nbreplicate to bpdm.
6. The replication checks are run and the replication starts. bpdm lets the source
storage server know when to initiate the replication. The source and target
storage servers then communicate to perform the actual replication of data.
7. bpdm reads the image information and sends it to bpdbm on the master server.
importmgr (master server, import manager logs within the 226 nbstserv log):
Storage lifecycle policy (SLP) and Auto Image Replication (A.I.R.) logging 136
SLP configuration and management
nbrmms (logs on the media server responsible for monitoring the disk storage):
stsem (storage server event manager, stsem logs within the 222 nbrmms log):
On the media servers that perform the duplication, view the appropriate bpdm and
bptm legacy logs. On the media server that initiates the A.I.R. replication operation
and on the media server that performs the subsequent import, you can view the
bpdm legacy log for additional details.
bpdm (verbose 5)
bptm (verbose 5)
You can increase the plugin logging to get additional details within bptm/bpdm or
the third-party vendors OST plugin log file regarding the duplication, replication,
and import operations.
On the master server, the following legacy logs are also helpful to review:
■ admin - (the admin log logs the bpduplicate or nbreplicate command for the
job)
■ bpdbm - (the NetBackup Database Manager program that contains backup policy
information, such as files, media, and client information)
nbstl –L –all_versions
To list the images that are under SLP control (that is, they are waiting for the
completion of their secondary operations), use the following command:
nbstlutil report
To display the SLP parameters using the CLI, the bpgetconfig command can be
run on the master server:
Storage lifecycle policy (SLP) and Auto Image Replication (A.I.R.) logging 137
SLP configuration and management
nbstlutil repllist
To list images that are pending an A.I.R. import into the target environment (run on
the target master server), use the following command:
nbstlutil pendimplist
Chapter 10
Snapshot technologies
This chapter includes the following topics:
■ VMware backup
The basic processing steps for snapshot creation and backup are the following (this
includes Windows open file backups that employ multiple data streams):
Snapshot technologies 140
Snapshot Client backup
5 bpfis contacts bprd to request transfer of bpfis state files from client to server.
This operation is enabled by default.
6 bprd requests bpcd on the client to send a list of bpfis state files.
7 bprd copies each state file from the client to the master.
8 bpfis sends snapshot information and completion status to bpbrm and exits.
bpbrm, in turn, reports the snapshot information and status to nbjm and exits.
nbjm relays the information and status to nbpem.
9 nbpem submits to nbjm a child job for the backup with a file list derived from
the snapshot information. nbjm starts bpbrm to back up the snapshot.
10 bpbrm starts bpbkar on the client. bpbkar sends the file catalog information to
bpbrm, which relays it to the NetBackup file database (bpdbm) on the master
server.
11 bpbrm starts the process bptm (parent) on the media server.
12 One of the following occurs: The next step depends on whether the media
server backs up itself (bptm and bpbkar are on the same host) or the media
server backs up a client that resides on a different host.
■ If the media server backs up itself, bpbkar stores the snapshot-based image
block-by-block in shared memory on the media server.
■ If the media server backs up a client that resides on a different host, the
bptm process on the server creates a child process of itself. The child
receives the snapshot-based image from the client by means of socket
communications and then stores the image block-by-block in shared
memory.
13 The original bptm process takes the backup image from shared memory and
sends it to the storage device (disk or tape).
Snapshot technologies 141
VMware backup
14 bptm sends the backup completion status to bpbrm, which passes it to nbjm.
15 When nbpem receives the backup completion status from nbjm, nbpem tells
nbjm to delete the snapshot. nbjm starts a new instance of bpbrm on the media
server, and bpbrm starts a new instance of bpfis on the client. bpfis deletes
the snapshot on the client, unless the snapshot is of the Instant Recovery type,
in which case it is not automatically deleted. bpfis and bpbrm report their status
and exit.
VMware backup
The following shows a VMware backup process.
The basic processing steps for a VMware backup operation are the following:
Snapshot technologies 142
VMware backup
10 The backup and archive manager (bpbkar) loads the Veritas Mapping Services
(VxMS) which loads the VMware Disk Development Kit (VDDK) APIs. The APIs
are used for reading from the vSphere datastore. VxMS maps the stream during
run-time and identifies the contents of the vmdk file. bpbkar uses VxMS to
send the file catalog information to bpbrm, which relays it to the database
manager bpdbm on the master server.
11 bpbrm also starts the process bptm (parent) on the media server.
The following shows the operation of the Veritas V-Ray within VxMS:
■ Veritas V-Ray within VxMS generates the catalog of all the files inside the
VMDK from both Windows and Linux VMs. The operation occurs while
backup data is being streamed. bpbrm on the media server sends this
catalog information to the master server.
■ The file system inode level also identifies unused and deleted blocks. For
example, if the application on VM allocates 1 TB of space for a file, of which
only 100 GB is currently used, the backup stream includes only that 100
GB. Similarly, if you delete a 1 TB file that was fully allocated in the past,
VxMS skips the deleted blocks (unless the blocks are now allocated for a
new file) from the backup stream. This optimization not only speeds up the
backup stream, but reduces needed storage even when deduplication is
not enabled.
■ If the source side deduplication feature is enabled, the VMware backup
host does the deduplication. The NetBackup deduplication plug-in using
the mapping information that VxMS generates and sees the actual files in
the file system within the VMDK. This V-Ray vision is established by the
NetBackup deduplication plug-in that loads a dedicated stream handler that
understands the VxMS mapping info.
■ Because these operations occur on the VMware backup host, the ESX
resources and the VM resources are not used. This setup is true off-host
backup with no burden on the production vSphere. Even the source side
deduplication occurs in an off-host system.
12 If the media server is the VMware Backup Host, bpbkar stores the
snapshot-based image block-by-block in shared memory on the media server.
If the media server is backing up a separate VMware Backup Host that is not
the media server, the bptm process on the server creates a child process of
itself. The child uses socket communications to receive the snapshot-based
image from the VMware Backup Host and stores the image block-by-block in
shared memory.
13 The original tape manager (bptm) process takes the backup image from shared
memory and sends it to the storage device (disk or tape).
Snapshot technologies 144
Snapshot backup and Windows open file backups
14 bptm sends backup completion status to bpbrm, which passes it to nbjm and
nbpem.
15 nbpem tells nbjm to delete the snapshot. nbjm starts a new instance of bpbrm
on the media server, and bpbrm starts a new instance of bpfis on the VMware
Backup Host. bpfis deletes the snapshot on the vSphere environment. bpfis
and bpbrm report their status and exit.
Figure 10-1 Snapshot backup and Windows open file backup using multiple
data streams
NetBackup user
Backup Policy
Configuration interface or
Management command line
Database
bpbackup or
File Database bpdbm nbproxy nbpem bprd
bparchive
nbproxy nbjm
EMM Database
bpbrm bpcd
Snapshot
Mount
Create
bpcd
bptm bptm
(parent) (child)
Mount
Tape
Request Control Path
Notes:
* For details on these components, see the Media and Device Management Functional Description
later in this chapter.
** If the media server is backing up itself (server and client on same host), there is no bptm child:
bpbkar sends the data directly to shared memory.
Snapshot technologies 146
Snapshot backup and Windows open file backups
A separate parent job creates all snapshots, then a child job backs up the snapshot.
The following sequence of operations is for snapshot creation and backup, including
the Windows open file backups that employ multiple data streams:
■ The NetBackup master server or primary client initiates the backup. This action
causes the NetBackup request daemon bprd to submit a backup request to the
Policy Execution Manager nbpem. nbpem processes the policy configurations.
■ nbpem (through nbjm) starts a parent job to create the snapshot. This job is
separate from the job that backs up the snapshot.
■ nbjm starts an instance of bpbrm through bpcd on the media server, and bpbrm
starts bpfis through bpcd on the client.
■ bpfis creates a snapshot of the client’s data by means of a snapshot method.
■ The next step depends on the following: Whether the media server backs up
itself (bptm and bpbkar on the same host), or the media server backs up a client
on a different host. If the media server backs up itself, bpbkar stores the
snapshot-based image block by block in shared memory on the media server.
If the media server backs up a client that resides on a different host, bptm on
the server creates a child process of itself. The child receives the snapshot-based
image from the client by means of socket communications and then stores the
image block-by-block in shared memory.
■ The original bptm process then takes the backup image from shared memory
and sends it to the storage device (disk or tape).
Information is available on how the tape request is issued.
See "Media and device management process" in the NetBackup Troubleshooting
Guide.
■ bptm sends backup completion status to bpbrm, which passes it to nbjm.
■ When nbpem receives backup completion status from nbjm, nbpem tells nbjm to
delete the snapshot. nbjm starts a new instance of bpbrm on the media server,
Snapshot technologies 147
Snapshot backup and Windows open file backups
and bpbrm starts a new instance of bpfis on the client. bpfis deletes the
snapshot on the client, unless the snapshot is of the Instant Recovery type, in
which case it is not automatically deleted. bpfis and bpbrm report their status
and exit.
For more information, see the NetBackup Snapshot Client Administrator’s Guide.
Note that Windows open file backups do not require Snapshot Client.
Chapter 11
Locating logs
This chapter includes the following topics:
■ acsssi logging
■ bpbackup logging
■ bpbkar logging
■ bpbrm logging
■ bpcd logging
■ bpcompatd logging
■ bpdbm logging
■ bpjobd logging
■ bprd logging
■ bprestore logging
■ bptm logging
■ daemon logging
■ ltid logging
■ nbemm logging
■ nbjm logging
■ nbpem logging
■ nbproxy logging
■ nbrb logging
Locating logs 149
acsssi logging
■ PBX logging
■ reqlib logging
■ robots logging
■ tar logging
■ vnetd logging
acsssi logging
On UNIX systems, the NetBackup ACS storage server interface (acsssi)
communicates with the ACS library software host.
How to access The acsssi process uses the legacy logging method. If
legacy debug logging is not enabled on your NetBackup
servers, you must create the appropriate directories for each
process.
See “About legacy logging” on page 36.
bpbackup logging
The bpbackup command-line executable is used to initiate user backups.
UNIX: /usr/openv/netbackup/logs/bpbackup
How to access The bpbackup process uses the legacy logging method. If
legacy debug logging is not enabled on your NetBackup
servers, you must create the appropriate directories for each
process.
bpbkar logging
The backup and archive manager (bpbkar) is used to read client data, which is
sent to the media server to write to the storage media. It also collects metadata
about the files that have been backed up to create the files file.
UNIX: /usr/openv/netbackup/logs/bpbkar
How to access The bpbkar process uses the legacy logging method. If
legacy debug logging is not enabled on your NetBackup
servers, you must create the appropriate directories for each
process.
See “About legacy logging” on page 36.
bpbrm logging
The NetBackup backup and restore manager (bpbrm) manages the client and bptm
process. It also uses the error status from the client and from bptm to determine
the final status of backup and restore operations.
UNIX: /usr/openv/netbackup/logs/bpbrm
How to access The bpbrm process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
bpcd logging
The NetBackup client service (bpcd) authenticates remote hosts and launches
processes on local hosts.
UNIX: /usr/openv/netbackup/logs/bpcd
How to access The bpcd process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
See “About legacy logging” on page 36.
bpcompatd logging
The NetBackup compatibility service (bpcompatd) creates connections between
some multi-threaded processes and NetBackup legacy processes.
UNIX: /usr/openv/netbackup/logs/bpcompatd
How to access The bpcompatd process uses the legacy logging method.
If legacy debug logging is not enabled on your NetBackup
servers, you must create the appropriate directories for each
process.
bpdbm logging
The NetBackup Database Manager (bpdbm) manages the configuration, error, and
file databases.
UNIX: /usr/openv/netbackup/logs/bpdbm
How to access The bpdbm process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
See “About legacy logging” on page 36.
bpjobd logging
The bpjobd service manages the jobs database and relays job statuses to the
Activity Monitor.
UNIX: /usr/openv/netbackup/logs/bpjobd
How to access The bpjobd process uses the legacy logging method. If
legacy debug logging is not enabled on your NetBackup
servers, you must create the appropriate directories for each
process.
See “About legacy logging” on page 36.
bprd logging
The NetBackup request daemon (bprd) responds to client and administrative
requests for backups, restores, and archives.
UNIX: /usr/openv/netbackup/logs/bprd
How to access The bprd process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
See “About legacy logging” on page 36.
bprestore logging
The bprestore command-line executable is used to initiate restores. It
communicates with bprd on the master server.
UNIX: /usr/openv/netbackup/logs/bprestore
How to access The bprestore process uses the legacy logging method.
If legacy debug logging is not enabled on your NetBackup
servers, you must create the appropriate directories for each
process.
See “About legacy logging” on page 36.
bptm logging
The NetBackup tape management process (bptm) manages the transfer of backup
images between the client and the storage device (tape or disk).
Locating logs 154
daemon logging
UNIX: /usr/openv/netbackup/logs/bptm
How to access The bptm process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
See “About legacy logging” on page 36.
daemon logging
The daemon log includes debug information for the Volume Manager service (vmd)
and its associated processes.
UNIX: /usr/openv/volmgr/debug/daemon
How to access The daemon log uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
See “About legacy logging” on page 36.
ltid logging
The logical tape interface daemon (ltid), also called the NetBackup Device
Manager, controls the reservation and assignment of tapes.
UNIX: /usr/openv/volmgr/debug/ltid
How to access The ltid process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
nbemm logging
On the server that is defined as the master server, the NetBackup Enterprise Media
Manager (nbemm) manages devices, media, and storage unit configuration. It supplies
nbrb with a cache list of available resources, and manages the internal state of
storage, (UP/DOWN) based on heartbeat information and disk polling.
UNIX: /usr/openv/logs/nbemm
How to access The nbemm process uses the unified logging method. Use
the vxlogview and vxlogmgr commands to view and
manage the unified log files.
See “About unified logging” on page 13.
nbjm logging
The NetBackup Job Manager (nbjm) accepts job requests from nbpem and from
media commands, and it acquires the necessary resources for the jobs. It interacts
with bpjobd to provide updates to the activity monitor states, starts the bpbrm media
manager service as needed, and updates the internal job states.
UNIX: /usr/openv/logs/nbjm
How to access The nbjm process uses the unified logging method. Use the
vxlogview and vxlogmgr commands to view and manage
the unified log files.
See “About unified logging” on page 13.
nbpem logging
The NetBackup Policy Execution Manager (nbpem) creates policy and client tasks
and determines when jobs are run.
UNIX: /usr/openv/logs/nbpem
How to access The nbpem process uses the unified logging method. Use
the vxlogview and vxlogmgr commands to view and
manage the unified log files.
See “About unified logging” on page 13.
nbproxy logging
The proxy service nbproxy enables nbpem and nbjm to query master server catalogs.
UNIX: /usr/openv/netbackup/logs/nbproxy
How to access The nbproxy process uses the legacy logging method. If
legacy debug logging is not enabled on your NetBackup
servers, you must create the appropriate directories for each
process.
nbrb logging
On the master server, the NetBackup Resource Broker (nbrb) locates logical and
physical resources from a cached list of resources to satisfy storage units, media,
and client reservations for jobs. It initiates drive queries every 10 minutes to check
the state of the drives.
UNIX: /usr/openv/logs/nbrb
How to access The nbrb process uses the unified logging method. Use the
vxlogview and vxlogmgr commands to view and manage
the unified log files.
See “About unified logging” on page 13.
UNIX: usr/openv/wmc/webserver/logs
UNIX: usr/openv/logs/nbwebservice
How to access The web services use the unified logging method for the web
applications. Use the vxlogview and vxlogmgr commands
to view and manage the unified log files.
See the NetBackup Troubleshooting Guide for more information on how to access
the web services logs.
install path\NetBackup\logs\nbatd
install_path\NetBackup\logs\nbcert
C:\ProgramData\Symantec\NetBackup\InstallLogs\
WMC_configureCerts_yyyymmdd_timestamp.txt
UNIX:
usr/openv/logs/nbatd
usr/openv/logs/nbcert
usr/openv/wmc/webserver/logs/configureCerts.log
How to access The nbcert and nbatd logs use unified logging. Use the
vxlogview and vxlogmgr commands to view and manage
the unified log files. The configureCerts.log uses a
simple logging style and not VxUL.
NetBackup creates the following logs when it renews the web server certificate.
Locating logs 159
PBX logging
install_path\NetBackup\logs\nbatd
install path\NetBackup\logs\nbwebservice
C:\ProgramData\Symantec\NetBackup\InstallLogs\
WMC_configureCerts_yyyymmdd_timestamp.txt
UNIX:
usr/openv/logs/nbatd
install_path\NetBackup\logs\nbwebservice
usr/openv/wmc/webserver/logs/configureCerts.log
How to access The nbwebservice (OID 466 and 484) and nbatd (OID
18) logs use unified logging. Use the vxlogview and
vxlogmgr commands to view and manage the unified log
files. The configureCerts.log uses a simple logging
style and not VxUL.
See “About unified logging” on page 13.
See the NetBackup Troubleshooting Guide for more information on how to access
the web services logs.
PBX logging
Private Branch Exchange (PBX) is the communication mechanism used by most
NetBackup processes.
UNIX: /opt/VRTSpbx/log
How to access The PBX process uses the unified logging method. Use the
vxlogview and vxlogmgr commands to view and manage
the unified log files. Note that the PBX product ID used to
access the unified log files differs from the NetBackup product
ID. The PBX product ID is 50936.
See the NetBackup Troubleshooting Guide for more information on how to access
PBX logs.
See “About backup logging” on page 67.
See “About restore logging” on page 95.
reqlib logging
The reqlib log includes debug information on the processes that request media
management services from EMM or the Volume Manager service (vmd).
UNIX: /usr/openv/volmgr/debug/reqlib
How to access The reqlib log uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
See “About legacy logging” on page 36.
robots logging
The robots log includes debug information on all robotic daemons, including the
txxd and txxcd daemons.
UNIX: /usr/openv/volmgr/debug/robots
How to access The robots log uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
tar logging
The Tape Archive program (tar) writes restore data to the client disk. On Windows
clients, the binary name is tar32.exe and on UNIX clients the binary name is nbtar.
How to access The tar process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
See “About legacy logging” on page 36.
Log location The txxd and txxcd processes do not have their own log
files. Instead, errors are logged in the robots debug log and
the system log. The system log is managed by syslog on
UNIX and by the Event Viewer on Windows.
See “About UNIX system logs” on page 10.
vnetd logging
The NetBackup Legacy Network Service (vnetd) is a communication mechanism
used to create firewall-friendly socket connections.
UNIX: /usr/openv/logs/vnetd or
/usr/openv/netbackup/logs/vnetd if the vnetd
directory exists there. If the vnetd directory exists in both
locations, logging occurs only in
/usr/openv/netbackup/logs/vnetd.
How to access The vnetd process uses the legacy logging method. If legacy
debug logging is not enabled on your NetBackup servers,
you must create the appropriate directories for each process.
■ Undo logging
1 2
bpjava-msvc nbatd
3A
2
Java login 4 bpjava-susvc 3B
Hashed security
tokens
bpjava-usvc
The following steps describe the Java-based administration console login process:
1. The user initiates a login request to the Java-based administration console.
The credentials are sent to bpjava-msvc over the Secure Sockets Layer (SSL)
using the Server Security Certificate.
2. The bpjava-msvc process authenticates the token through nbatd, which reads
the hashed security tokens on the server.
3. The following steps describe the process with the session certificate:
■ The bpjava-msvc process sends a response to the console login with a
session token and a fingerprint of the session certificate.
■ The bpjava-msvc process initiates the appropriate bpjava-*usvc process
and the session certificate and token are passed to one of the following
processes:
■ bpjava-susvc for the NetBackup Administration Console
Note: The following processes are used: bpjava-msvc, which controls the login
and authentication; bpjava-susvc, which is the administration console process;
and bpjava-usvc, which is the client Backup, Archive, and Restore (BAR) interface.
1. The user initiates a login to the console. The credentials are sent to
bpjava-msvc over the SSL (using the Server Security Certificate).
2. The bpjava-msvc process authenticates the user who uses the user credentials
that were received in step 1.
3. After the user is authenticated, the bpjava-msvc process performs the following:
■ Generates the entities that are called the self-signed session certificate,
the key, and the session token.
■ Launches the daemon bpjava-*usvc to gather more requests from the
NetBackup Java-based administration console.
■ Passes the self-signed session certificate and the session token to
bpjava-*usvc.
■ Sends the session token and the fingerprint of the session certificate to the
Java-based administration console.
■ Persists session token and user information to a secure directory
(install_path/var; for example, usr/openv/var) in a file on the
NetBackup host. This directory is accessible only to the root/administrator.
The file name format is as follows:
hash(session token)_bpjava-*usvc_pid
Java-based administration console logging 166
Setting up a secure channel between the Java-based administration console and either nbsl or nbvault
4. bpjava-*usvc uses the session certificate to start the secure channel with the
Java-based administration console. This secure channel is a one-way
authenticated SSL channel. (Only the server certificate is present and there is
no peer certificate. There is no certificate from the Java-based administration
console side.)
5. The Java-based administration console receives the session certificate as a
part of the initial SSL handshake. It verifies the authenticity of the session
certificate by using the pre-existing fingerprint of the session certificate (see
step 3). The Java-based administration console calculates the fingerprint of
the session certificate that was received from bpjava-*usvc due to the SSL
handshake. It compares the new fingerprint with the fingerprint sent by msvc.
6. Once the authenticity of the certificate is verified, the Java-based administration
console sends the session token (received in step 3) to bpjava-*usvc.
7. bpjava-*usvc verifies the received session token with the pre-existing one
(see step 3).
8. The success of the session token validation creates trust between
bpjava-*usvc and the Java-based administration console.
hash(session token)_susvc_pid
Java-based administration console logging 167
Setting up a secure channel between the Java-based administration console and either nbsl or nbvault
8. The success of the verification creates a trust between nbsl/nbvault and the
Java-based administration console.
9. All further communication occurs between nbsl/nbvault and the Java-based
administration console on this trusted secure channel.
Java-based administration console logging 168
Java-based administration console logging configuration on NetBackup servers and clients
C:\Program Files\Veritas\NetBackup\logs\user_ops\nbjlogs
install_path\Veritas\Java\setconf.bat
3. Locate the following line (line 12 down from the top of the file) and remove the
remark (REM) to activate it, as shown in the following example:
■ Change line from: REM SET NB_INSTALL_PATH=C:\\Program
Files\\Veritas\\NetBackup
C:\Program Files\Veritas\NetBackup\logs\user_ops\nbjlogs
printcmds=true
debugMask=0x00040000
1. Gather the following Java console logs from the following pre-existing log
directories on the system from which the GUI was started:
■ UNIX: /usr/openv/netbackup/logs/user_ops/nbjlogs
■ Windows: install directory\netbackup\logs\user|ops\nbjlogs
■ /usr/openv/netbackup/logs/bpjava-msvc
■ /usr/openv/netbackup/logs/bpjava-susvc
■ /usr/openv/netbackup/logs/bpjava-usvc
ADMIN_VERBOSE = 5
BPJAVA-MSVC_VERBOSE = 5
BPJAVA-SUSVC_VERBOSE = 5
BPJAVA-USVC_VERBOSE = 5
Java-based administration console logging 170
Configuring and gathering logs when troubleshooting Java GUI issues
■ install_dir\VERITAS\NetBackup\logs\bpjava-msvc
■ install_dir\VERITAS\NetBackup\logs\bpjava-susvc
■ install_dir\VERITAS\NetBackup\logs\bpjava-usvc
ADMIN_VERBOSE = 5
BPJAVA-MSVC_VERBOSE = 5
BPJAVA-SUSVC_VERBOSE = 5
BPJAVA-USVC_VERBOSE = 5
6. Run the following commands to set up detailed nbatd (OID 18) and nbsl (OID
132) logging. OID 137 (NetBackup libraries) and OID 156 (CORBA/ACE) write
to the caller that requires access to either the libraries or CORBA/ACE, as
follows:
vxlogcfg -a -p NB -o 18 -s DebugLevel=6
vxlogcfg -a -p NB -o 132 -s DebugLevel=6
vxlogcfg -a -p NB -o 137 -s DebugLevel=6
vxlogcfg -a -p NB -o 156 -s DebugLevel=6
7. Gather the nbatd and nbsl logs located in the following directory paths:
For UNIX:
■ /usr/openv/logs/nbsl
■ /usr/openv/logs/nbatd
For Windows:
■ install_dir\VERITAS\NetBackup\logs\nbsl
■ install_dir\VERITAS\NetBackup\logs\nbatd
Undo logging
Ensure that you undo the logging after you have gathered the logs that relate to
your troubleshooting issue.
To remove the log configuration settings, use the following commands:
vxlogcfg -r -p NB -o 18 -s DebugLevel=6
vxlogcfg -r -p NB -o 132 -s DebugLevel=6
vxlogcfg -r -p NB -o 137 -s DebugLevel=6
vxlogcfg -r -p NB -o 156 -s DebugLevel=6
On the master server, comment out the following Java VERBOSE entries in the
bp.conf file (UNIX) or in the registry (Windows):
■ ADMIN_VERBOSE
■ BPJAVA-MSVC_VERBOSE
■ BPJAVA-SUSVC_VERBOSE
■ BPJAVA-USVC_VERBOSE
Index
A bparchive
acssel, description 81 log 38, 40
acsssi logging 149 bpbackup
acsssi, description 81 log 38, 41
admin log 43 bpbackup logging 149
administration interface BPBACKUP_POLICY 67
activity logging 59 BPBACKUP_SCHED 67
errors 57 bpbkar
Application Event log 54 log 38, 41
application server status codes (Java interface) 58 logging 150
ascd, description 81 bpbrm 146
Auto Image Replication (A.I.R.) 132 log 43
import process flow 135 logging 150
logging 135 bpcd
process flow logging 134 server log 43
avrd, description 82 UNIX client log 38, 41
bpcd logging 151
bpcompatd logging 151
B bpdbjobs log 43
backup bpdbm
and archive processes 66 log 44
and restore startup process 65 logging 152
logging 62, 67 bpdm log 43
NetBackup catalogs 103 bpfis 146
procedure, basic 63 bphdb
process 62 log 39
deduplication 118 BPINETD 94
multiplexing 67 bpinetd log 40
snapshot overview 144 bpinetd.log 40
synthetic processes 107 bpjava-*usvc process 164
UNIX clients 67 bpjava-msvc log 44, 60
Backup, Archive, and Restore (BAR) interface 164 bpjava-msvc process 164
backup_tape log 39 bpjava-susvc process 164
barcode operations 77 bpjava-usvc log 60
basic backup procedure 63 bpjobd logging 152
bin bplist
media and device management 80 log 39, 41
bjava-usvc process 164 bpmount
bp log 39, 41
UNIX client log 38 bporaexp log 39
bp.conf bporaexp64 log 39
file 67 bporaimp log 39
Index 173
O rotation (continued)
OpenStorage Technology (OST) of logs 24
configuration 126 unified logging 16
configuration and management 129
operating system errors 58 S
originator IDs SAN client backup procedure 99
list of 17 SAN client backup process flow 99
originatorID SAN Client Fiber Transport backup 98
unified logging 16 SAN Client Fiber Transport restore 101
Secure Sockets Layer (SSL) 164
P sending backup logs 69
PBX logging 159 server
Private Branch Exchange (PBX) 123 NetBackupdebug logs 37
process descriptions Server Security Certificate 164
NetBackup 65 session certificate 164
product ID for NetBackup 16 set retention limits for logs 54
productID setting up a secure channel
unified logging 16 between the Java console and bpjava-* 165
between the Java console and either nbsl or
nbvault 166
Q Shared Storage Option management process 75
query string 27 snapshot
backup process overview 146
R Snapshot backup 144
raw partitions Snapshot Client backup 138
restore process 92 Snapshot Client backup procedure 140
reports SSO. See Shared Storage Option
NetBackup 9 startup
reqlib logging 160 NetBackup 65
restore logging 95 startup process 71
restore logs media and device management 71
send to Veritas Technical Support 96 Status Collection Daemon 38
restore procedure from disk 91 stderr 57
restore procedure from tape 90 stdout 57
restore process 88 storage lifecycle policy (SLP) 132
UNIX client 92 configuration and management 136
Windows client 94 duplication process flow 133
retention logging 135
of logs 25 synthetic backup
retention limits for logs logs 110
setting 54 synthetic backups 107
robot drive selection 73 syslogd 10
robotic system logs 10
control daemons 72
daemons 72 T
robots logging 160 tape
RolloverMode 36 restore from 90
rotation
legacy logging 48
Index 177