0% found this document useful (0 votes)
272 views

Rotating Diagnostic and Administration Notification Logs

DB2 rotating diagnostics

Uploaded by

Mevro
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
272 views

Rotating Diagnostic and Administration Notification Logs

DB2 rotating diagnostics

Uploaded by

Mevro
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

New Feature in the DB2 V9.

7 –
Rotating Diagnostic and
Administration Notification Logs

applies to:
All SAP releases running DB2 9.7 for Linux, Unix and Windows. For more information, visit the Landscape
Design and Architecture homepage.

Summary
This new feature - rotating diagnostic and administration notification logs – was introduced in DB2 9.7 for
Linux, UNIX and Windows. It allows customers to define the maximum amount of space to be used for
diagnostic and administration notification logs and therefore prevents the overflow of the filesystem specified
by the database manager configuration parameter DIAGPATH.

Author: Anatoly Malchevsky


Company: SAP Canada Inc.
Created on: 01 June 2009

Author Bio
Since joining SAP in 1999, Anatoly Malchevsky has been a member of the IBM SAP Integration
and Support Centre located in the Toronto IBM Lab. He currently tests, develops and integrates
new features of DB2 LUW with SAP. Through his SAP development support role and being a
Customer Advocate for IBM, Anatoly assists SAP consultants and customers with activities such
as troubleshooting and performance optimization

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 1
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

Table of Contents
Overview ............................................................................................................................................................. 3 
Working with rotating diagnostic logs ................................................................................................................. 3 
Enabling rotating diagnostic logs. ................................................................................................................... 3 
Disabling rotating diagnostic logs ................................................................................................................... 5 
Tools support for rotating diagnostic logs ....................................................................................................... 7 
Best practices ..................................................................................................................................................... 7 
Related Content ................................................................................................................................................ 10 
Copyright........................................................................................................................................................... 11 

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 2
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

Overview
IBM DB2 diagnostic logs contain information about the operation of the database. There are two types of
logs: db2<instance>.nfy - intended for use by database and system administrators and db2diag.log,
intended for use by DB2 Support.
In releases, prior to IBM DB2 9.7, logs were allowed to grow indefinitely. If diagnostic logs are not archived,
eventually they might overflow the filesystem and DB2 will loose its ability to log diagnostic information. In a
worst case scenario , when the logs reside on the same filesystem as other DB2 vital data, DB2 itself will fail
to function properly. Furthermore, if db2diag.log was never pruned it can contain diagnostic data for several
years of operation and in a service event all the data will be transferred via ftp to SAP/IBM which can take a
long time and therefore increase the response time of a service message.
Huge db2diag.log overflowing customer file systems and slowing down problem determination due the large
amount of data which has to be transferred is a paint point for both customers and service personnel.
The new solution will also ease the daily workload of DBA’s.

Working with rotating diagnostic logs


Enabling rotating diagnostic logs.
To prevent the situation described above, a new feature was introduced in IBM DB2 9.7 - rotating diagnostic
and administration notification logs. A new database manager configuration parameter DIAGSIZE allows
customer to predefine how much space can be used for all diagnostic and notification logs.
The default value for DIAGSIZE is 0. It corresponds to the old behavior, i.e. only one db2diag.log and one
db2<instance>.nfy logs are used and both files can grow indefinitely.
Example of the update database manager configuration command:

db2 update dbm cfg using DIAGSIZE 1024

DIAGSIZE is measured in Megabytes, the minimum value for the DIAGZISE parameter is 2.
The maximum DIAGSIZE value can not exceed the free disk space under DIAGPATH. An SQL error
SQL5130N will be returned if DIAGSIZE value exceeds the free disk space.
Example of the attempt to set DIAGSIZE to a value exceeding the available free space.

saplp1:db2lr1 59> db2 update dbm cfg using DIAGSIZE 13000


SQL5130N The value specified for the configuration parameter "diagsize" is
not in the valid range of "2MB" to "12245MB(free space)".

DIAGSIZE is not a dynamic parameter; the new value/behaviour will take effect only after an instance restart.
Once DIAGSIZE is set to a non-zero value and the instance is restarted, the new logging behavior is
adopted:
• Diagnostic messages will be written to a series of files, i.e. db2diag.0.log … db2diag.###.log. The
total number of logs will not exceed 10.
• The administration notification logs will be written to a series of files, i.e. db2<instance>.0.nfy …
db2<instance>.###.nfy. Total number of logs will not exceed 10.
• The total size of the db2diag.###.log and db2<instance>.###.nfy files will not exceed the DIAGSIZE
value in MB.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 3
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

Here’s an example of using the DIAGSIZE,


By default, DIAGSIZE is set to 0, and if we issue ls –l in the db2dump directory, we see only db2diag.log and
db2lr1.nfy (SAP system id is R1):

saplp1:db2lr1 88> ls -l
total 8696
-rw-rw-rw- 1 db2lr1 dblr1adm 3362 2009-06-03 09:21 db2diag.log
-rw-r----- 1 db2lr1 dblr1adm 6291312 2009-04-16 10:11 db2eventlog.000
-rw-rw-rw- 1 db2lr1 dblr1adm 1074 2009-06-03 09:21 db2lr1.nfy
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:47 events
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:46 stmmlog

Then, we issue command

saplp1:db2lr1 85> db2 update dbm cfg using DIAGSIZE 2


DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed

And restart the instance.

saplp1:db2lr1 88> ls -l
total 8696
-rw-rw-rw- 1 db2lr1 dblr1adm 2781 2009-06-03 09:46 db2diag.0.log
-rw-rw-rw- 1 db2lr1 dblr1adm 3362 2009-06-03 09:21 db2diag.log
-rw-r----- 1 db2lr1 dblr1adm 6291312 2009-04-16 10:11 db2eventlog.000
-rw-rw-rw- 1 db2lr1 dblr1adm 1362 2009-06-03 09:46 db2lr1.0.nfy
-rw-rw-rw- 1 db2lr1 dblr1adm 1074 2009-06-03 09:21 db2lr1.nfy
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:47 events
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:46 stmmlog

Two new files appeared in the db2dump directory, db2diag.0.log and db2lr1.0.nfy
The partition ratio between the db2diag.###.log files and the db2<instance>.###.nfy files is approximately
90% to 10%. 90% of the DIAGSIZE value is assigned to the db2diag.###.log files and 10% to the
db2<instance>.###.nfy files.
Back to our example, after a while, DB2 fills up db2diag.0.log and creates db2diag.1.log, db2diag.2.log and
so on:

saplp1:db2lr1 88> ls -l
total 8696
-rw-rw-rw- 1 db2lr1 dblr1adm 172781 2009-06-03 10:02 db2diag.0.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172839 2009-06-03 10:07 db2diag.1.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172716 2009-06-03 10:12 db2diag.2.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172360 2009-06-03 10:18 db2diag.3.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172862 2009-06-03 10:23 db2diag.4.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172635 2009-06-03 12:07 db2diag.5.log
-rw-rw-rw- 1 db2lr1 dblr1adm 93338 2009-06-03 14:16 db2diag.6.log
-rw-rw-rw- 1 db2lr1 dblr1adm 3362 2009-06-03 09:21 db2diag.log
-rw-r----- 1 db2lr1 dblr1adm 6291312 2009-06-03 14:16 db2eventlog.000
-rw-rw-rw- 1 db2lr1 dblr1adm 2436 2009-06-03 14:16 db2lr1.0.nfy
-rw-rw-rw- 1 db2lr1 dblr1adm 1074 2009-06-03 09:21 db2lr1.nfy
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:47 events
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:46 stmmlog

Rotating diagnostics logs would never fill up the db2dump directory due to a DB2 mechanism, called “early
totaling ability”

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 4
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

When creating a new log file, DB2 always checks the available disk space, and if there is no enough space
left for creating a new log file and if there are at least two rotating logs, DB2 will delete the oldest log file even
if it hasn’t reached the size limit yet ( the number of rotating logs hasn’t reached 10 yet), so that DB2 will get
some space to create the new log file.

Disabling rotating diagnostic logs


To get back to the old logging behavior, DIAGSIZE has to be set to 0 and the instance restarted
In our example:
saplp1:db2lr1 82> db2 update dbm cfg using DIAGSIZE 0
DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed
successfully.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 5
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

Contents of the db2dump directory:


saplp1:db2lr1 88> ls -l
total 8696
-rw-rw-rw- 1 db2lr1 dblr1adm 172781 2009-06-03 10:02 db2diag.0.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172839 2009-06-03 10:07 db2diag.1.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172716 2009-06-03 10:12 db2diag.2.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172360 2009-06-03 10:18 db2diag.3.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172862 2009-06-03 10:23 db2diag.4.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172635 2009-06-03 12:07 db2diag.5.log
-rw-rw-rw- 1 db2lr1 dblr1adm 93338 2009-06-03 14:36 db2diag.6.log
-rw-rw-rw- 1 db2lr1 dblr1adm 3362 2009-06-03 14:36 db2diag.log
-rw-r----- 1 db2lr1 dblr1adm 6291312 2009-06-03 14:36 db2eventlog.000
-rw-rw-rw- 1 db2lr1 dblr1adm 2436 2009-06-03 14:36 db2lr1.0.nfy
-rw-rw-rw- 1 db2lr1 dblr1adm 1074 2009-06-03 09:21 db2lr1.nfy
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:47 events
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:46 stmmlog
After setting DIAGSIZE to 0, DB2 resumes logging to files db2diag.log and db2lr1.nfy.
If we decide to go back to the rotating logging and set DIAGSIZE to 2 and restart the instance, loggings
resumes to the latest rotating logs - files db2diag.6.log and db2lr1.0.nfy
saplp1:db2lr1 88> ls -l
total 8696
-rw-rw-rw- 1 db2lr1 dblr1adm 172781 2009-06-03 10:02 db2diag.0.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172839 2009-06-03 10:07 db2diag.1.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172716 2009-06-03 10:12 db2diag.2.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172360 2009-06-03 10:18 db2diag.3.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172862 2009-06-03 10:23 db2diag.4.log
-rw-rw-rw- 1 db2lr1 dblr1adm 172635 2009-06-03 12:07 db2diag.5.log
-rw-rw-rw- 1 db2lr1 dblr1adm 157454 2009-06-03 15:21 db2diag.6.log
-rw-rw-rw- 1 db2lr1 dblr1adm 34353 2009-06-03 14:36 db2diag.log
-rw-r----- 1 db2lr1 dblr1adm 6291312 2009-06-03 14:36 db2eventlog.000
-rw-rw-rw- 1 db2lr1 dblr1adm 2436 2009-06-03 14:36 db2lr1.0.nfy
-rw-rw-rw- 1 db2lr1 dblr1adm 1074 2009-06-03 14:36 db2lr1.nfy
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:47 events
drwxrwxr-t 2 db2lr1 dblr1adm 4096 2009-06-03 09:46 stmmlog

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 6
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

Tools support for rotating diagnostic logs


DB2 tools like db2support and db2diag and the table UDF PD_GET_DIAG_HIST (used by the SAP
transaction DB6COCKPIT) support rotating diagnostic logs.
User might use the db2diag tool for reading and filtering db2 diagnostic records. There is no change in the
output format as such seen by the user but they should be aware that all rotating diagnostic logs are
considered by the db2diag tool if the DIAGSIZE parameter is set a non-zero value in memory. So, a
command like “db2diag” will read all rotating db2 diagnostic logs if DIAGSIZE parameter is set to a non-zero
value in memory.
db2diag –f (follow mode) will also support the new feature. db2diag –f will read the latest rotating db2
diagnostic file in use. So, let’s say we start db2diag –f, it will start reading db2diag.0.log file. Then, once
db2diag.1.log file is created, we switch over to db2diag.1.log file in follow mode

Best practices
DB2 9.7 documentation recommends setting DIAGSIZE to a value greater then 50 MB, but not to more then
80% of the free space of the db2dump directory. To prevent the loss of the diagnostic information required
for analyzing a problem, I would recommend setting DIAGSIZE to 1024 (1GB). That will allow DB2 to create
up to ten 10MB notification logs and up to ten 90MB diagnostic logs. Average size of a db2diag.log,
(depending on the customer’s workload ) in a SAP system is between 2 and 4 MB per week. So, if we “set
aside” 1 GB of space for the diagnostic logs, it should cover situations when the database is in distress and
warning/error messages are written at a high rate.
Here’s a table containing db2diag.log sizes in byte per week collected during the support process. This is a
real-life data demonstrating the average and the maximum db2diag.log sizes encountered by the support.
2009-04-26_2009-05-03_db2diag.log 1,927,908
2009-05-03_2009-05-10_db2diag.log 702,930
2009-01-15_2009-01-18_db2diag.log 130,323
2009-01-19_2009-01-25_db2diag.log 871,886
2009-01-04_2009-01-11_db2diag.log 1,115,355
2009-01-12_2009-01-18_db2diag.log 560,668
2008-08-13_2008-08-17_db2diag.log 239,851
2008-08-18_2008-08-24_db2diag.log 226,092
2008-08-25_2008-08-31_db2diag.log 18,516,542
2008-09-01_2008-09-07_db2diag.log 170,534
2008-09-08_2008-09-14_db2diag.log 127,592
2008-09-15_2008-09-21_db2diag.log 147,959
2008-09-22_2008-09-28_db2diag.log 193,475
2008-09-29_2008-10-05_db2diag.log 2,472,403
2008-10-06_2008-10-12_db2diag.log 3,286,843
2008-10-13_2008-10-19_db2diag.log 2,454,594
2008-10-20_2008-10-26_db2diag.log 2,412,210
2009-02-16_2009-02-22_db2diag.log 1,652,670
2009-02-23_2009-03-01_db2diag.log 1,547,413
2009-03-02_2009-03-08_db2diag.log 1,825,581
2009-03-09_2009-03-15_db2diag.log 1,716,638
2009-03-16_2009-03-22_db2diag.log 1,710,216
2009-03-23_2009-03-29_db2diag.log 1,678,891
2009-03-26_2009-03-29_db2diag.log 371,870
2009-01-03_2009-01-04_db2diag.log 182,687,172
2009-01-05_2009-01-11_db2diag.log 19,029,678
Customers who like to keep a version of db2diag.log for a longer period can archive the logs once a week to
a different filesystem/directory then db2dump, i.e. do not save information into subdirectories under the
db2dump. You should not allow the db2dump directory to grow to an unmanageable size and be cluttered

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 7
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

with outdated information. It will increase the size of the db2support output file and it will make problem
analysis harder due to the high data volume. The db2diag tool has an option to archive diagnostic logs; it will
also process rotating logs appropriately.
Here’s an example:

saplp1:db2lr1 94> db2diag -A 20090608


db2diag: Moving "/db2/LR1/db2dump//db2diag.22.log"
to "20090608/db2diag.22.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.23.log"


to "20090608/db2diag.23.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.24.log"


to "20090608/db2diag.24.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.25.log"


to "20090608/db2diag.25.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.26.log"


to "20090608/db2diag.26.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.27.log"


to "20090608/db2diag.27.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.28.log"


to "20090608/db2diag.28.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.29.log"


to "20090608/db2diag.29.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.30.log"


to "20090608/db2diag.30.log_2009-06-08-07.55.56"

db2diag: Moving "/db2/LR1/db2dump//db2diag.31.log"


to "20090608/db2diag.31.log_2009-06-08-07.55.56"

Another option would be to use a simple shell/batch file to archive the contents of the db2dump directory.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 8
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

For example,

#!/bin/ksh
sid=${USER##db2}
typeset -u SID="$sid"
ts=`date +%Y%m%d%H%M`

print "Archiving contents of the db2dump to /db2/$USER/db2dumps/${ts?}"


mkdir /db2/$USER/db2dumps/${ts?}

mv /db2/$SID/db2dump/db2diag*.log /db2/$USER/db2dumps/${ts?}
mv /db2/$SID/db2dump/$USER.nfy /db2/$USER/db2dumps/${ts?}
mv /db2/$SID/db2dump/stmmlog /db2/$USER/db2dumps/${ts?}
mv /db2/$SID/db2dump/events /db2/$USER/db2dumps/${ts?}

The script should be executed as a user db2<sid> and the directory /db2/db2<sid>/db2dumps should be
created beforehand.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 9
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

Related Content
IBM DB2 Database for Linux, UNIX, and Windows Information Center
For more information, visit the Landscape Design and Architecture homepage.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 10
New Feature in the DB2 V9.7 – Rotating Diagnostic and Administration Notification Logs

Copyright
© 2009 SAP AG. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG.
The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries,
zSeries, System i, System i5, System p, System p5, System x, System z, System z9, z/OS, AFP, Intelligent Miner, WebSphere,
Netfinity, Tivoli, Informix, i5/OS, POWER, POWER5, POWER5+, OpenPower and PowerPC are trademarks or registered trademarks of
IBM Corporation.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems
Incorporated in the United States and/or other countries.
Oracle is a registered trademark of Oracle Corporation.
UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of
Citrix Systems, Inc.
HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts
Institute of Technology.
Java is a registered trademark of Sun Microsystems, Inc.
JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by
Netscape.
MaxDB is a trademark of MySQL AB, Sweden.
SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All
other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves
informational purposes only. National product specifications may vary.
These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP
Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or
omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the
express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an
additional warranty.
These materials are provided “as is” without a warranty of any kind, either express or implied, including but not limited to, the implied
warranties of merchantability, fitness for a particular purpose, or non-infringement.
SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may
result from the use of these materials.
SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these
materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and
does not endorse your use of third party web pages nor provide any warranty whatsoever relating to third party web pages.
Any software coding and/or code lines/strings (“Code”) included in this documentation are only examples and are not intended to be
used in a productive system environment. The Code is only intended better explain and visualize the syntax and phrasing rules of
certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for errors
or damages caused by the usage of the Code, except if such damages were caused by SAP intentionally or grossly negligent.

SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com


© 2009 SAP AG 11

You might also like