TL Tips 2
TL Tips 2
When a device is detected on a given HBA port a device name (e.g. rmt0) will be assigned to it.
AIX subsequently ensures that always the same device is mapped to this name, across system
restarts or SAN reconfigurations.
This way all possible changes are "transparent" to the application using the device. You don't need
Qlogic's SanSurfer or Emulex' lputil at all.
The above is true as long as device definitions are not deleted from the ODM database,
i.e. as long as the "-d" flag of "rmdev" or "KEEP definition in database = No" of "smitty rmdev"
are not used.
There might be rare cases where device definitions must be completely removed from ODM,
maybe due to a major device driver upgrade. I, personally, faced such a situation only once, when
upgrading Atape from Ver.7 to Ver. 9, if I remember well.
In such cases you'll have to review the device definitions in your backup application in order to
check for consistency with the OS names.
You might be interested in further details about "Managing device addressing of SAN attached
tape".
It deals mainly with tape addressing in a TSM environment, but has some useful general
information too.
Problem
Requirements for using persistent binding with the NBU Shared Storage Option(SSO)
Solution
Veritas NBU 6.0 Media Manager System Administrator's Guide for Windows, Page: 270
Veritas NBU Shared Storage Guide UNIX, Windows, Linux Release 6.5, Page: 58
Ensuring proper tape drive configuration in a SAN environment is critical to the proper operation
of NBU SSO. Often, SAN hardware is conf igured to use automatic device discovery. This practi
ce works well with disk storage devices, but is not recommended for tape storage sub-systems wh
ere persistent hardware device configuration or identification is necessary.
The Windows operating system detects the SAN hardware SCSI devices via a Fibre HBA. This de
tection phase then stores or maps SCSI drive information in the Windows registry.
The Windows operating system must properly detect and map all SCSI devices in the registry bef
ore NBU device configuration. In NBU device configuration, SCSI device information is gathered
from the operating system and placed into the NBU device database. The database stores the exact
SCSI location of every device detected by the HBA.
When the SAN hardware is configured for automatic detection, any change in the SAN system wi
ll trigger the operating system to re-scan the HBA and remap the devices in the registry. In such a
n event, the new tape device mapping can present a conflict within the NBU device database. As a
result, NBU will no longer communicate with the proper tape device and device erro rs will occur.
Example:
NBU may attempt to mount media into a tape device from the previous configuration. NBU is con
figured to mount media into drive 1, but after a hardware re-scan the device is mapped to drive 2.
This new configuration, although detected by the operating system, is not automatically updated in
NBU. This will cause the backup to fail.
A second example is when NBU queries the SCSI device and cannot sense the device on the bus.
This condition will result in a missed drive signal stating that the "device is not ready" or that it is
"in use" which can be interpreted as a hardware problem, and will down the drive.
A third example in this situation is that a media eject may not occur and is presented to the operato
r as a hardware problem. This condition may interrupt NBU processes and ultimately NBU will re
fuse to release the drive for further NBU drive operations.
To counteract the effect of SAN Hardware automatic device detection, it is important to ensure th
at consistent SCSI device path and Logical Unit Number (LUN) mapping information remains con
sistent even after a reboot.
Below are some examples of HBA configurations when using Emulex LP8000 series cards.
a) Clearing the Automatically Map SCSI Devices check box will prevent the SCSI device path
information from changing. Changes in the SCSI device path information could conflict with
other SAN equipment and result in tape devices going down or cause interruption to backup
or restore operations.
b) Clearing the Automatic LUN Mapping check box will allow proper configuration of your ma
pped SCSI IDs to the appropriate WWN of the HBA. Differences in LUN mapping informat
ion can als o cause problems similar to those caused by inconsistencies in the SCSI device pa
ths listed above.
LP9000 series HBAs share a similar configuration utility. The rules of persistent binding, LUN ma
pping, and SCSI device path consistencies vary between HBA manufacturers, but the need for con
sistent hardware mapping information is essential for proper communication.
Below is an excerpt from Emulex which illustrates the advantages of persistent binding and useful
ness in backups directed to fibre attached tape devices:
A Fibre Channel target is assigned its WWNN at loop initialization time; the SCSI target ID for th
at target is assigned by the device driver when the device is firstly discovered. It is possible for th
e WWNN to change between one loop initialization and the next. Every time a system boots or a t
arget is added to or removed from the Fibre Channel, the loop will be re-initialized. After a system
has booted, it will maintain a constant view of the same target ID because the driver software rem
aps the SCSI target ID to the new WWNN on the fly.
In an environment that utilizes persistent binding, once a system has booted, it will maintain a con
stant view of the same target ID because the driver software remaps the SCSI target ID to the new
WWNN on the fly.
A second system may use a different SCSI target ID for that target. Thus, an administrator seeking
to work with the same target across multiple hosts must be prepared to encounter the situation whe
re the same Fibre Channel target is known by different SCSI target IDs. Use persistent binding to
maintain a consistent target ID on all systems. A common scenario for this type of situation would
be a network administrator who needs to consistently map to a tape drive for backup purposes. "
Unix SA performed reconfiguration reboot to fix disk issue on NBU media server. After reb oot,
we found all tape drives were visible at OS leve ( cfgadm ) however, NBU SA was miss ing half
of tapes in tapes configuration. Lateron, NBU SA had to perform tapes reconfigurat ion to get all
tapes back to availability.
We understand during reboot, hardware device id got changed for few of tapes and hence impacte
d tapes configuration for NBU.
How can we make device id persistent for tapes during reboot so that we dont face this issue on n
ext reboot or reconfig reboot ?
Persistance binding is required at OS level to make sure that the Drive paths are not changing...
You would need to check the related OS technotes about how to make a persistant binding.
On solaris you need to modify the devlink tab (/etc/devlink.tab) so that the /dev/rmt/# remains the
same for specific tapes.
Eg of devlink.tab:
type=ddi_pseudo;name=sg;addr=w20010013214436b6,2a; sg/c\N0t\A1l42
First of all, check if broken tape device links existed at that time. It sometimes happen that broken
link is removed during reconfiguring boot and link name for actual devices become adjusted.
For example, when /dev/rmt/0 is broken link and /dev/rmt/1 is link for actual device, broken link
will be removed and link for actual device is renamed to /dev/rmt/0 during reconfiguration. This
lead mismatch between OS configuration and NBU device managenet. So it is very importa nt to
keep device links clean anytime.
You need to first of all determine which make/model HBA is used for tape access.
Configure Persistent Binding using hba tool or by editing the /kernel/drv/xxxx.conf. You can find
details on the web site of HBA manufacturer (e.g. Qlogic or Emulex).
I see /etc/devlink.tab file is existing. Latest one is created on Feb 14 after reboot time. Is there
possibility, even if devlink.tab exists, reconfig reboot can affect tape configuration ?
As per below output, I see there was devlink.tab before reboot, then why it impacted tapes to miss
after reboot ?
hardware configuration exactly, exactly, same with before? Any change on cabling, HBA bus slot,
SCSI ID, WWN? Using 3rd party FC HBA without persistent binding?
If yes(completely same), I doubt broken rmt links generated by hardware configuration change wi
thout reconfigure boot.
BTW, How "ls -l /dev/rmt" looks like?
Even though the /etc/devlink.tab exists, it would not by default contain the rules for tape devices,
and hence the devfsadm command which runs while reconfiguration reboot will automatically cre
ate links for tape devices. For the binding to be persistant and not to change after a reconfiguration
reboot, you need to add the rules manually for each tape device in the devlink.tab.
Below is some output from server rmt devices and devlink file. Does it say there is no rule defined
for tapes ? I see a lot of entries with wwn in it.
/#> ls -al /dev/rmt/4
lrwxrwxrwx 1 root root 77 Feb 13 19:10 /dev/rmt/4 ->
../../devices/pci@780/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0/st@w500507630f5bf614,0:
If I follow below link for creating rules for persisting binding on tapes, where can I find currently
defined tape numbers to set in rules in devlink file?
This approach can easily lead to duplicate links in /dev/rmt. Any tapes discovered before entries w
ere specified in /etc/devlink.tab have automatically created links. When entries are added and dev
fsadm is run, the original links remain in /dev/rmt, resulting in duplicate links. To remove the orig
inal links in /dev/rmt, run the rm /dev/rmt/* command before running devfsadm.
This approach cannot be used with multiple-port tape drives that are attached to multiple HBAs. If
multiple HBAs are attached to the same tape LUN, the system detects two tape drives instead of
one. The one that appears last in the prtconf output gets the link generated by the /etc/devlink.tab.
The following example shows a sample entry for tape in the devlink.tab file.
type=ddi_byte:tape;addr=PWWN,LUN-number; rmt/rmt-number\M0
Change the rmt # to whatever /dev/rmt/N is required. Then change the PWWN/LUN to match the
desired tape device. You can obtain this value by running the ls -l cmd on the existing /dev/rmt/
link as shown below.
# ls -l /dev/rmt/4
lrwxrwxrwx 1 root root 69 Oct 6 14:57 /dev/rmt/4 ->
../../devices/pci@1f,700000/SUNW,qlc@2/fp@0,0/st@w5005076300617717,0:
For example, you wanted the /dev/rmt/<n> to be 40, you would create an entry in /etc/dev/link.
tab like the following :
# type=ddi_byte:tape;addr=w5005076300617717,0; rmt/40\M0
You can then add this line to the devlink file on every Solaris server on the SAN that uses this dri
ve so that it always appears as minor node 40.
Configuration Steps
1). Create the entries in /etc/devlink.tab as described in Creating Tape Links.
If devfsadm has previously discovered the devices, you must determine the device address by runn
ing the ls -l command on the existing link.
Note: Be sure to assign /dev/rmt/ N numbers to avoid conflicts with any automatically configured
devices, as described above.
2). Remove existing links from /dev/rmt by running the rm /dev/rmt/* command.
3). Run devfsadm, this command creates new links as per the entries in /etc/devlink.tab in addition
to automatically creating links for any unspecified devices.
Some tape libraries can require an annual service, and the term for this is PM, or 'Preventative Ma
intenance' - whereby the mechanical gearing of a library robot arm is cleaned and re-greased, and
then re-aligned. An engineer performing PM shoudl also inspect for worn gears, worn motors, lo
ose drive belts, replace air-filters, and clean any light residual traces of grease, grime and dust fro
m within the inside of the tape library, and clean the bar-code laser and 'eye/lens'.
Some libraries can be rebooted so that, as part of their reboot process, they use special (hidden fro
m NBU) bar-coded points within the library chassis to re-align themselves internally, so that the ro
bot arm lines up correctly with the format of the tape drives - this process is sometimes referred to
a s 'Library learn' sequence.
So more modern libraries are more advanced and no longer require any annual PM, as they are sel
f lubricating, and self adjusting.
ioctl is usually a hardware error, though slightly odd its happened on multiple drives at the same
time. If these drives are all connected via the sam HBA it could be an HBA issue (or firmware).
You should also consider the cables and even the drives.
You could rebuild the sg driver, but to be honest, I'm not convinced this will be at fault., usless the
drives have disappeared from the output of /usr/openv/volmgr/bin/scan, and even then it doesn't
mean its the sg driver. But if thyey have disappeared and you rebuild the sg driver and they don't
reappear, then it's pretty certain it's something else.
What happens if you run the mt command, eg.
mt -f <device file> stat
You could also use /usr/openv/volmgr/bin/scsi_command -d
/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0
To rebuild the sg driver
modunload -i $(echo $(modinfo |grep "sg (SCSA" |awk '{print $1}'))
mv /kernel/drv/sg.conf /kernel/drv/sg.conf.old
/usr/openv/volmgr/bin/driver/sg.install
ls -l /dev/IBMtape*
crw-r--r-- 1 root root 251, 3071 Jan 31 01:35 /dev/IBMtape
crw------- 1 root root 251, 0 Jan 31 01:35 /dev/IBMtape0
crw------- 1 root root 251, 1024 Jan 31 01:35 /dev/IBMtape0n
crw------- 1 root root 251, 1 Jan 31 01:35 /dev/IBMtape1
crw------- 1 root root 251, 1025 Jan 31 01:35 /dev/IBMtape1n
crw------- 1 root root 251, 2 Jan 31 01:35 /dev/IBMtape2
crw------- 1 root root 251, 1026 Jan 31 01:35 /dev/IBMtape2n
# use the following (remove # marks in the lines below and add # marks in the above lines)
# if petro driver parameter is ON
#n=''
#if [ $minor -gt $PETRONONREWOFFSET -a $minor -lt $PETRONONRESVOFFSET ] ; then
# n='n'
#elif [ $minor -gt $PETRONONRESVOFFSET -a $minor -lt $CHANGEROFFSET ] ; then
# NAME='IBMtapenrsv'
#fi
# lin_tape does not have the IOCTL that scsi_id uses however it
# has a sysfs entry pointing to the generic device which does support that IOCTL
SYMLINKPATH="$SYSFS$DEVPATH/device/generic"
if [ ! -d $SYMLINKPATH ] ; then
exit 1
fi
cd $SYMLINKPATH
GDEVICEPATH="`pwd -P|sed -e '{ s/\/sys\(.*\)/\1/ }'`"
# use the following line if you want to use WWNNs for tape drives
# and serial numbers for medium changers
# TAPE_ID="`/sbin/scsi_id -gus $GDEVICEPATH`"
WWN="`/sbin/scsi_id -gus $GDEVICEPATH`"
#use the following (remove # mark in the line below and add # in the above line)
# if you want to use serial number for tape drives and medium changers
TAPE_ID="`/sbin/scsi_id -p 0x80 -gus $GDEVICEPATH`"
echo "${WWN} ${TAPE_ID} ${DEVPATH}" >> /tmp/tape-name.txt
echo -n $TAPE_ID-$NAME$n
When creating definitions for the drives in TSM you have to add the /lin_tape/by-id to the path
like this.....
/dev/lin_tape/by-id/SIBM_03584L32_00000*******401-IBMchanger
/dev/lin_tape/by-id/SIBM_ULT3580-TD5_000*******-IBMtape
Notes: the *'s are my serial numbers so you will obviously have different numbers there.
Our tape drives names something like /dev/ibm2109, where this represents drive 09 of frame 1 of
IBMLIB2. This helps us quickly map a drive to its location as well as have common and persistent
names on all servers.
/etc/udev/scripts/tapedrive_name.sh
#!/bin/bash
# Work with udev for IBM SCSI tape device persistent naming support
# by Alex Iribarren $Revision: 1.2 $ $Date: 2009/10/16 16:43:25 $
. /etc/tapenames.conf
SYSFS=/sys/class/lin_tape/
DEVPATH=$1
# lin_tape does not have the IOCTL that scsi_id uses however it
# has a sysfs entry pointing to the generic device which does support that IOCTL
SYMLINKPATH="$SYSFS$DEVPATH/device/generic"
if [ ! -d $SYMLINKPATH ] ; then
exit 1
fi
cd $SYMLINKPATH
GDEVICEPATH="`pwd -P|sed -e '{ s/\/sys\(.*\)/\1/ }'`"
# use the following line if you want to use WWNNs for tape drives
# and serial numbers for medium changers
TAPE_ID="`/sbin/scsi_id -gus $GDEVICEPATH`"
PREFIX=${TAPE_ID:0:15}
FRAME=${TAPE_ID:15:1}
ROW=${TAPE_ID:16:1}
LIBRARY=`name $PREFIX`
FRAME=$(( FRAME + 1 ))
ROW=$(( 0x$ROW ))
if [ ${#ROW} == 1 ]; then
ROW=0$ROW
fi
echo ${LIBRARY}${FRAME}${ROW}
exit 0
/etc/udev/scripts/tapechanger_name.sh
#!/bin/bash
# Work with udev for IBM SCSI tape device persistent naming support by Alex Iribarren
# $Revision: 1.1 $ $Date: 2009/11/23 16:43:16 $
. /etc/tapenames.conf
SYSFS=/sys/class/lin_tape/
DEVPATH=$1
# lin_tape does not have the IOCTL that scsi_id uses however it
# has a sysfs entry pointing to the generic device which does support that IOCTL
SYMLINKPATH="$SYSFS$DEVPATH/device/generic"
if [ ! -d $SYMLINKPATH ] ; then
exit 1
fi
cd $SYMLINKPATH
GDEVICEPATH="`pwd -P|sed -e '{ s/\/sys\(.*\)/\1/ }'`"
# use the following line if you want to use WWNNs for tape drives and serial numbers for
# medium changers
TAPE_ID="`/sbin/scsi_id -gus $GDEVICEPATH`"
SERIAL=${TAPE_ID:14:16}
name ${SERIAL}
exit 0
/etc/tapenames.conf
#!/bin/bash
function name {
case $1 in
'0000078A01400402') # for old ibmlib0, '0000078A01400401')
ret='ibmlib0'
;;
'3500507630f2442') # for old ibmlib0 '3500507630f0073' | '3500507630fc073')
ret='ibm0'
;;
'0000000A17470401')
ret='ibmlib2'
;;
'3500507630f1467')
ret='ibm2'
;;
*)
logger -p kern.err -t udev "Unable to find library for $1 (WWNN: $TAPE_ID)"
exit 1
esac
echo $ret
}
Problem(Abstract)
After the lin_tape device driver is upgraded, the lin_tape persistent naming devices are removed
from the Linux system.
Cause
Linux device persistent naming is done via Linux udev. In earlier versions of lin_tape device
driver udev rule file /etc/udev/rules.d/98-lin_tape.rules and script /sbin/udev.get_lin_tape_id.sh
were provided to create the lin_tape persistent devices.
In newer versions of lin_tape device driver, the lin_tape rule file and script used to create the persi
stent naming devices are removed, as a result, after the upgrade of lin_tape device driver, the persi
stent naming devices will not be recreated.
Persistent naming on Linux is controlled by Linux "udev". After a successful lin_tape install, a lin
_tape udev rule file will be created at the location: /etc/udev/rules.d/98-lin_tape.rules.
This rule file needs to be edited to create the lin_tape persistent names. The following steps may
be used.
1. Find out the device serial or WWN attribute of the tape drive.
Run the udevinfo command for each lin_tape or /dev/IBMtapex device.
For example, using /dev.IBMtape1 :
#udevinfo -a -p 'udevinfo -q path -n /dev/IBMtape1'
...
looking at parent device '/devices/pci0000:00/0000:00:09.0/0000:09:00.0/host2/rport-2:0-2/targ
et2:0:0/2:0:0:1':
ID=="2:0:0:1"
BUS=="scsi"
DRIVER=="lin_tape"
SYSFS{primary_path}=="NA"
SYSFS{ww_port_name}==""
SYSFS{ww_node_name}=="0x20020000C9D93932"
SYSFS{serial_num}=="111411000"
...
Depending on the Linux kernel, the udevinfo may not be available. In this case, use udevadm. For
example :
udevadm info --attribute-walk --name /dev/IBMtape1
...
ATTR{serial_num}=="1114110004"
ATTR{ww_node_name}=="0x20020000C9D93932"
ATTR{ww_port_name}=="0x10000000C9D93932"
ATTR{primary_path}=="NA"
ATTR{sys_encryption_proxy}=="1"
ATTR{sys_encryption_write}=="2"
3: Reload udev:
udevadm control --reload-rules
5. Define the drive and the drive path using the device persistent name. For example :
DEFINE DRIVE LIBRARY_Name DRIVE1 SERial=AUTOD element=AUTOD Online=YES
DEFINE PATH SERVER_NAME DRIVE1 SRCT=server DESt=drive
LIBRARY=LIBRARY_Name device=/dev/lin_tape/by-id/lin_tape1_111411 online=yes