IBM SAP Implementation and Operation in Integrated Environments Version 1.

IBM SAP Technical Brief

SAP Implementation and Operation in Integrated Environments

The IBM Blue Stack Proof of Concept combines IBM System x, XIV and DB2 on Linux and VMware to provide superior business performance and reduced operational effort

IBM SAP International Competence Center Walldorf, Germany

Version: 1.01 Status: October 08th, 2010

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 1 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Preface Edition Notice (August 2010) This is the first edition of this document. Scope This document is intended for IBM personnel, Business Partners and Customers involved in the implementation of an SAP infrastructure on IBM System x, VMware, IBM XIV, IBM DB2 and Linux. The authors of this document are Irene Hopf Maik Gasterstaedt Markus Fehling Paul Henter Elke Hartmann-Bakan All from the IBM SAP International Competence Center (ISICC) in Walldorf, Germany. The project team (listed alphabetically) are: IBM: Markus Fehling, ISICC Maik Gasterstaedt, ISICC Patrick Hartman, SAP LinuxLab Elke Hartmann-Bakan, ISICC Paul Henter, ISICC Irene Hopf, ISICC (Project Lead) SAP Center of Excellence / Value Prototyping: Abhik Bose, Project Lead (Application, Performance) Juergen Hamm, Project Lead (Infrastructure, Performance) Ingo Dahm, SAP Basis Consultant Mohamed Beradhi, PI Consultant Frank Leggewie, SAP Basis Consultant

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 2 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Acknowledgments go to: Alexander Bruder, SAP CoE Alan Cummings, eTS UK Bernd Eberhardt, IBM SWG, Rational Torsten Fellhauer, SAP CoE Thorsten Hepting, SAP CoE Michael Hesse, VMware Jan Mnch, IBM ITS Peter Rupp, ISICC And the whole project team for patience and good spirit during the project. Sponsors are: Wolfgang Knobloch, IBM SAP Alliance Sales Growth Program SWG Manager Nina Johannsen, IBM STG Worldwide SAP Business Development Manager Laurent Montaron, IBM SAP Enablement on IBM Systems Manager Tag Robertson, IBM System x and BladeCenter SAP Solutions Michelle Tidwell, IBM System Storage ISV Enablement Manager Feedback We are interested in any feedback you have. Please send your comments to the IBM SAP International Competency Center (ISICC) Infoservice [email protected] hotline. We can been contacted via telephone at: +49 (0) 6227 73 1099

Disclaimer This document is subject to change without notification and will not comprehensively cover the issues encountered in every customer situation. It should be used only in conjunction with the product literature accompanying the products listed above. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS.

Edition history Version 1.0: 31st Aug. 2010, Final version Version 1.01: 08th Oct. 2010, minor typos corrected, summary changed

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 3 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Table of Contents
1 2 IBM SYSTEM X BLUE STACK FOR SAP APPLICATIONS EXECUTIVE SUMMARY................ 5 INTRODUCTION ....................................................................................................................................... 9 2.1 2.2 3 USE CASES............................................................................................................................................ 9 SCOPE ................................................................................................................................................. 10

OPERATIONAL SETUP.......................................................................................................................... 11 3.1 3.2 HW CONFIGURATION ......................................................................................................................... 11 SW CONFIGURATION (RELEASE COMBINATION)................................................................................. 15

FUNCTIONAL SETUP FOR THE BUSINESS PROCESSES ............................................................. 16 4.1 4.2 4.3 4.4 BUSINESS PROCESS SCENARIO OVERVIEW ......................................................................................... 16 ERP & PI CENTRIC SCENARIOS 1 & 2................................................................................................. 17 PORTAL CENTRIC SCENARIOS 3, 4 & 5................................................................................................ 18 BW CENTRIC SCENARIO 6 .................................................................................................................. 19

VIRTUALIZATION MEASUREMENTS WITH REALISTIC WORKLOAD.................................. 20 5.1 5.2 5.3 5.4 5.5 5.6 5.7 VIRTUALIZATION SETUP ..................................................................................................................... 21 THROUGHPUT TESTS ON TWO X3950 M2 ............................................................................................ 21 CPU UTILIZATION .............................................................................................................................. 22 MEMORY UTILIZATION ....................................................................................................................... 26 I/O BEHAVIOR ON IBM XIV STORAGE ............................................................................................... 29 CCMS MONITORING IN TRANSACTION OS07N ................................................................................... 31 THROUGHPUT TESTS ON ONE X3850 X5 ............................................................................................. 33

BUSINESS RESILIENCY ........................................................................................................................ 37 6.1 6.2 6.3 6.4 6.5 CONCEPTS EVALUATED IN THIS PROOF OF CONCEPT .......................................................................... 38 INTEGRATED DB2 BACKUP WITH TSM FOR ERP ................................................................................ 39 DATA PROTECTION WITH IBM FLASHCOPY MANAGER (FCM) .......................................................... 40 VMWARE SITE RECOVERY MANAGER ................................................................................................ 41 HOW TO REDUCE THE TIME FOR PLANNED OUTAGES ......................................................................... 52

7 8

CLONING.................................................................................................................................................. 53 IBM DB2 .................................................................................................................................................... 54 8.1 8.2 8.3 8.4 STORAGE LAYOUT:............................................................................................................................. 54 CONCURRENT I/O PARAMETER SETTINGS........................................................................................... 54 INSTALLATION PARAMETERS .............................................................................................................. 55 SAP DBA COCKPIT FOR DB2 LUW................................................................................................... 58

REFERENCES .......................................................................................................................................... 59 9.1 VMWARE............................................................................................................................................ 59 VMWARE SITE RECOVERY MANAGER .............................................................................................................. 59 9.2 DB2.................................................................................................................................................... 59 9.3 TSM ................................................................................................................................................... 60

10

APPENDIX (A).......................................................................................................................................... 61 10.1 10.2 10.3 10.4 10.5 10.6 10.7 10.8 10.9 10.10 A: RATIONAL PERFORMANCE TESTER (RPT) SETUP........................................................................... 61 A: RATIONAL PERFORMANCE TESTER RESULT SCREENS ON TWO X3950 M2 ..................................... 74 A: RATIONAL PERFORMANCE TESTER RESULT SCREENS ON ONE X3850 X5 ...................................... 76 A: VMWARE VSPHERE VIRTUAL CENTER .......................................................................................... 78 A: XIV: LAYOUT AND NAMING CONVENTION .................................................................................... 83 A: NETWORK INFRASTRUCTURE ......................................................................................................... 84 A: BACKUP / RECOVERY / RESTORE.................................................................................................... 85 A: DETAILS ON CLONING PROCEDURE ................................................................................................ 97 A: DB2 CONFIGURATION .................................................................................................................. 108 A: SAP INSTANCE AND DEFAULT PROFILES ..................................................................................... 115 Page 4 of 120

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

1 IBM System x Blue Stack for SAP applications Executive Summary

Many organizations rely on integrated SAP solutions to run almost every aspect of their business operations rapidly and efficiently. Typically, the SAP applications at the heart of the business are mission-critical, and without them, enterprise operations can be severely affected. It is therefore essential to ensure that underlying IT infrastructure is capable of providing the necessary performance, productivity and reliability to support the SAP landscape. Issues such as system complexity and management play a very significant role in forming a suitable infrastructure strategy that will meet the business need. Many customers wish to improve the IT services relied on by their business functions, helping to reduce cost and manage the risk in their data centers. These challenges can be addressed by creating a dynamic infrastructure with a set of interconnected and optimally configured hardware and software products. In a joint project, SAP and IBM executed a Proof of Concept (PoC) to show the business advantages of taking an integrated end-to-end enterprise infrastructure approach with an Intel processor-based strategy. Called IBM Blue Stack for SAP applications, the PoC demonstrated how an integrated hardware and software Blue Stack using IBM and SAP components and infrastructure virtualization complements SAP Business Suite applications, delivering lower system complexity, easier management, and reduced costs. The Blue Stack PoC delivers tangible value for many customers running SAP applications through typical real world proof points demonstrating: Successful implementation of an integrated end-to-end IBM stack Reduced SAP infrastructure capacity requirements and increased system utilization through virtualization High Availability scenarios for protecting mission-critical information Reduced business risk with system resiliency and disaster recovery capabilities The PoC emulated a typical real-world scenario (see Figure 1 below), with a core set of SAP applications: SAP ERP, SAP NetWeaver BW, SAP NetWeaver Portal, SAP NetWeaver Process Integration and SAP Solution Manager. The business processes covered purchase order scenarios, self-service transactions via the portal and queries against the business warehouse.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 5 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Solution Manager

EP Prd passive (cluster)

Virtual Center

SAP NW71

SAP CRM

ECC Prd

ECC QA

IBM DB2, Tivoli Storage Manager VMware 4.0 vSphere Linux IBM System X

IBM DB2, Tivoli Storage Manager VMware 4.0 vSphere Linux IBM System X

BW Prd

FC SAN Switch Ethernet Switch

Rational Performance Tester

IBM XIV Storage Systems

Figure 1: IBM Blue Stack provides the storage, processor, switching and systems management that support reliable, high-performance SAP applications in the virtualized Linux environment To enable an SAP multi-application environment, the PoC used the Intel processor-based IBM System x servers, IBM XIV Storage System technologies (depicted in Figure 2), IBM DB2 Optimized for SAP, IBM Tivoli Storage FlashCopy Manager, IBM Rational Performance Tester for SAP solutions complemented by Novell SUSE Linux Enterprise Server and VMware vSphere components.

Figure 2: Overview of the Technology Landscape

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 6 of 120

Virtual Center

SAP NW70

ECC Dev

PI Prd

SAP SRM

EP Prd

IBM Stack

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Use Case Results The IBM and SAP team built a realistic datacenter environment to simulate the business processes of a customers common SAP landscape. They structured the following use cases around typical IT department operational tasks: 1. Virtualization of the SAP application landscape: Several SAP applications in separate virtual machines (VMs) spread across two physical servers. The team consolidated a set of realistic SAP workloads onto a virtualized environment, monitored system behavior, and reported system load, demonstrating consolidation (as illustrated in Figure 3) and significantly increased system utilization. 2. Monitoring of the SAP Landscape: Using standard tools from VMware, IBM, and SAP, the team monitored system behavior of the OS, DB, CPU, and memory as well as storage I/O rates. Simulation of realistic business process workloads was achieved using IBM Rational Performance Tester. 3. Disaster Recovery: VMware Site Recovery Manager and the IBM XIV Storage System were used to recover from a site failure. 4. High Availability: VMware VMotion and the IBM XIV storage system demonstrated easy and fast relocation of the SAP applications onto a back-up server to provide continuous operation during hardware maintenance. 5. High performance back up: Demonstrated high performance SAP application back-up in minutes using IBM XIV and IBM Tivoli Storage FlashCopy Manager for SAP production database back-up and restores. 6. Rapid SAP database cloning: Using IBM XIV Storage System snapshots and VMware vSphere, the SAP database instance was cloned in minutes. The project describes how to utilize that clone and separate virtual machine.

BW CRM EP CPU Utilization NW 70 NW 71 PI ECC Prd ECC QA ECC Dev SolMan SRM

Execution Time

Figure 3: Consolidated CPU utilization on one IBM System x3850 X5, each SAP application running in a separate VM IBM SAP International Competence Center, Walldorf, Germany Page 7 of 120
2010 Global Alliance Solutions, SAP

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

The Blue Stack Proof of Concept shows that the IBM infrastructure stack offers a uniquely integrated, easy to manage and virtualized platform to implement, protect and support multiple mission-critical SAP applications. IBM is the only technology vendor able to provide the complete end-to-end solution including software, server, storage, and network, combined with comprehensive management tools along with IBM services. The IBM technology delivers the business and technical benefits of an integrated infrastructure, well suited to integrated SAP application environments.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 8 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

2 Introduction
Many customers wish to improve the IT services provided to their business functions, helping to reduce cost and manage the risk in their data centers. This can be addressed by creating a dynamic infrastructure with a set of interconnected and optimal configured hardware and software products. IBM created a Proof of Concept to show our experiences and recommendations running a socalled blue stack of IBM hardware and software components which deliver that dynamic infrastructure for a typical SAP Business Suite landscape. The result of the Proof of Concept is this whitepaper, which is meant to help customers, business partners and IBM personnel gain insight and gather information for real-world implementations on IBM Systems hardware and software running the SAP Business Suite of applications. 2.1 Use Cases The IBM team structured use cases around day-to-day IT department operational tasks and built a realistic datacenter environment to simulate the business processes of a customers typical SAP landscape. The components of the SAP Application landscape that run on our blue stack infrastructure are: SAP ERP, SAP NetWeaver BW, SAP EP, SAP PI and SAP Solution Manager. The SAP landscape uses the IBM DB2 database running on SUSE Linux Enterprise Server, virtualized using VMware ESX. The entire software stack runs on IBM System x enterprise servers and IBM XIV storage. We also refer to several possible usages of other IBM software from Tivoli and Rational. The specific use cases covered in this whitepaper show various aspects IT departments use in their day to day business activities. We show the benefits of using virtualization to run several SAP applications in separate virtual machines (VMs) across two physical servers. We simulate realistic business process workloads with the IBM Rational Performance Tester. The ability to monitor the system behavior of the OS, DB, CPU and memory as well as storage I/O rates was accomplished using standard tools from VMware and SAP. To demonstrate business resiliency, we used VMware Site Recovery Manager and the IBM XIV storage system to address high availability and disaster recovery scenarios. We used IBM FlashCopy Manager in order to create FlashCopy-based backups of the SAP production database. We test cloning functionality using the IBM XIV storage system and describe the proper set up and use of cloning scenarios. In this project, we decided not to show an SAP upgrade process, as this is out of scope and not dependent on the specific setup covered in this PoC.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 9 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

2.2 Scope In this whitepaper, we pay attention to those test cases which are dependent on specific components of the selected infrastructure stack. If best practices or standard procedures are available, well documented and familiar, we use them as a basis of this Proof of Concept and do not try to recreate these operational tasks by hand. This whitepaper targets at SAP customers, neither particularly small nor particularly large. We are aware that every single SAP implementation is unique; still there are common approaches in every implementation that will achieve a flexible and dynamic infrastructure. The document contains three main parts. The executive summary comprises the first chapter. Chapters 2 through 8 describe the use cases, tests and results of the Proof of Concept. Lastly, chapters 9 and 10 give references and an appendix, which contains a detailed description of how the Proof of Concept was set up.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 10 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

3 Operational Setup
3.1 HW Configuration The following sections provide configurations of the server hardware, the network topology, the virtualization setup and the storage subsystems for the IBM Infrastructure in this PoC. 3.1.1 Server HW: 2 x IBM System x3950 M2 (7141-3SG): 4 x quad-core Xeon E7330 2.4GHz processors Main memory System IBM1 68GB main memory System IBM2 80GB main memory 1 Host Bus Adapter (HBA) & 2 onboard Network Interface Cards (NICs) 1 internal 146.8GB SAS HDD

Figure 4: IBM System x3950 M2 IBM System x3850 X5 - (7145-AC1): 4 x 8-core Xeon 7560 series 2.27GHz processors 256GB memory 1 HBA & 2 NICs 1 x 10Gbps Ethernet 4 internal 146GB SAS HDDs

Figure 5: IBM system x3850 X5

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 11 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

BladeCenter-E Chassis (8677 - 3XU ) 1 x Management Module 2 x Qlogic Fiber Channel Switch Module 2 x Cisco Gigabit Ethernet SM 5 x BladeServer LS41-7972 - 7AG

BladeCenter

10

11

12

13

14

Figure 6: BladeCenter-E Chassis The BladeCenter is equipped with two network switches. The first switch is used for the production network. The second switch is used for management and redundancy for the production network. We separate the hardware management portion of the BladeCenter management and the ESX server management from the ESX server network interfaces on the virtual networks switches. The network for production use is physically separated from the management network.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 12 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 7: Network topology Each server node is connected to the lab network using one Gigabit Ethernet adapters. The second Gigabit Ethernet adapter is not used in this environment. The web based interfaces to the Integrated Management Module (IMM in x3850X5) and Advanced Systems Management module (ASM in x3950 M2) allow remote management of the servers and give an overview of the system status. The ASM and the IMM are connected to the console switch network for management access. All systems are connected to the storage network via a 4 Gbps HBA, either via its own HBA or the IBM BladeCenter HBA switch.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 13 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

3.1.2 Virtualization Setup: The following Figure 8 shows the schematic layout of the test landscape:

Figure 8: Schematic System Layout The idea of the distribution of the applications over the landscape is to mix and match different systems as much as possible to achieve well-balanced utilization over the landscape. The PoC ran two test sets, first with the two 3950s and then all applications consolidated onto IBM x3850 X5. 3.1.3 Storage: IBM XIV The Proof of Concept used an IBM XIV 2810 Model A14 with 79TB of usable storage arranged over 180 physical disks. The IBM XIV storage system contains the volumes for the PoC landscape in a separate pool. Storage pools are used to control the storage resources of volumes and snapshots. They provide logical separation, like a folder. Within this pool three to four volumes were created for each virtual machine: - an operating system volume for the operating system - a database volume for the SAP application, database and log files for the test systems - a data volume for the SAP application and database files for the production systems - a log volume for the log files for the production systems - a swap volume for the operating systems swap partition. The volumes are mapped to each of the virtual machines as raw devices.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 14 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

In addition, the PoC uses a shared filesystem (here vmfs_shared, which uses vmfs V 3.33) accessed by all IBM System x hosts and all virtual machines in order to use VMware VMotion. IBM recommends setting the upper limit of the volume size to 2048 GB. In the appendix, find the detailed storage layout on the IBM XIV storage system (chapter 10.5). To ensure that all volumes are automatically visible to the host when a new host is added to the cluster, create a cluster on the IBM XIV storage system and assign all IBM System x hosts to this cluster. 3.2 SW Configuration (Release combination) The following sections describe the software configurations for the PoC. 3.2.1 Virtualization VMware (vSphere 4.0.1) with the SRM (site recovery manager) plug-in 3.2.2 Database IBM DB2 V9.7 FP1 including table and index compression for all systems 3.2.3 Operating system SUSE Linux Enterprise Server (SLES) version 10 SP3 3.2.4 SAP Application Components All SAP applications run as a 2-tiered installation with one SAP SIDDB (DB2) and the central instance SAP Business Suite ECC 6.0 EHP 4 installed on one VM running Linux (Production) SAP Business Suite ECC 6.0 EHP 4 installed on one VM running Linux (Quality Assurance) SAP Business Suite ECC 6.0 EHP 4 installed on one VM running Linux (Development) SAP BW 7.01 ABAP installed on one VM running Linux SAP EP 7.01 installed on one VM running Linux SAP PI 7.11 installed on one VM running Linux SAP Solution Manager 7.0 installed on one VM running Linux The SAP profile parameters are available in the appendix (chapter 10.10). 3.2.5 Load Simulation and Monitoring IBM Rational Performance Tester version 8.1.1 VMware vCenter Server with vManage version 4.01 Storage monitoring using the XIV GUI version 2.4.2 SAP monitoring using SAP Solution Manager and transaction OS07n across all managed Linux guests in the SAP landscape Wherever possible, we use the built-in monitoring functionality of the installed software components . Note: IBM Tivoli Storage Manager (TSM) is not installed for this PoC. There are no dependencies on TSM with a virtualized setup that require further explanation outside of the TSM installation guides. However we provide recommendations on the use of TSM in the chapter covering business resiliency (Appendix 10.7). IBM SAP International Competence Center, Walldorf, Germany Page 15 of 120
2010 Global Alliance Solutions, SAP

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

4 Functional Setup for the Business Processes

Our objective is to build a realistic customer scenario. It is not uncommon to find dozens of SAP systems in a datacenter. Thus, the PoC used the following SAP applications: SAP ERP, SAP NetWeaver BW, SAP PI, SAP EP and SAP Solution Manager. These applications cover the business processes used in this scenario. We intentionally did not focus on a specific industry because the PoC provides an example for a broad range of customer cases, both in terms of size of the business as well as industry sectors. Subsequently we selected an order-to-cash business process which is known to be used widely across multiple industries, regardless of whether a customers business is discrete manufacturing, life sciences or transportation. To reflect the customer of today, we configure the SAP ERP application as a three-system landscape with development, quality assurance and production systems. The other applications (BW, PI and EP) are only installed and customized once to reduce effort during the test procedures. However, to reflect a realistic customer scenario, we ran other SAP application systems independently from the test landscape on the same machine in order to show the robust scalability of the components using the architecture. 4.1 Business Process Scenario Overview The functional system landscape looks like this:
Rational Performance Tester
Webservice SAPgui http

PI

ERP

EP

BW

IDOC

http

http

Trusted RFC

Trusted RFC

Trusted RFC

Solution Manager (ABAP)

Figure 9: Overview of the functional system landscape and communication A real-time customer production environment is simulated by virtual users in the IBM Rational Performance Tester (RPT). One thousand employees are created in the backend ERP system using RPT. The same 1,000 employees are also created as SAP users on the system
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 16 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

through RPT. These SAP users are linked to the backend employees through RPT and used to perform all the following business process scenarios. RPT will run purchase orders (PO) and process the delivery of goods. Concurrently, RPT simulates users who log into EP to execute three iViews triggering standard BW reports about the purchase order transaction data. Thirdly, other business processes, a business trip request and the recording of work hours, are executed within the employee and manager self service (ESS, MSS) portal in parallel. All scenarios execute simultaneously to generate load in different systems with workload peaks at different times. This is meant to simulate a peak run in a customer business. Each simulated user is represented by a file on the OS level in the RPT systems. Following are detailed descriptions of each scenario. 4.2 ERP & PI Centric Scenarios 1 & 2
RPT
1 Create PO Script HTTP (IDOC XML) 1:1 messages IDOC Adapter (PORDCR) Create PO

PI

ERP

2 Create Goods Receipt Script SAP GUI Recording Create Goods Receipt

Figure 10: All scenarios that will be directly executed on the ERP system. When the users start to create purchase orders in a foreign (external, maybe non-SAP) system they are pushed to PI first and PI grows in terms of resource requirement. Thus, the server is utilized heavily by PI while the other applications in other VMs use less capacity. At first, the connection between PI and ERP is blocked, yet as soon as the connection is opened, the ERP system starts to process the purchase orders and subsequently consumes more resources. ERP is considered the central system for the customer daily business operations, so maximum load is generated on this system. As soon as the orders are processed, the delivery of these orders is triggered with a time delay. This business process returns and finally informs the end users, who ordered the goods, that the orders are being shipped and in process of delivery. Scenario 1 Single IDOC XML message is sent to PI by RPT, PI then maps it to the SAP IDOC for creating orders in ERP system and send it to ERP. ERP creates the order with an ERP external order number. One user in RPT is corresponds to one purchase order (PO) in ERP. Scenario 2 RPT simulates the logging in of users on the SAPGUI to create goods receipts. One user in RPT corresponds to one goods receipt in ERP.
Page 17 of 120

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

An external number range is used for the purchase orders being generated in the ERP system via PI & Portal, because the goods receipt process requires a PO number. Goods receipts are created only after the successful creation of the POs. 4.3 Portal Centric Scenarios 3, 4 & 5
RPT
3 HTTP (Web Recording) Portal Scripts 4 HTTP (Web Recording) Portal Scripts Create Time Recording HTTP (Web Recording) Portal Scripts Create PO Create BTR Standard Interface Business Content (ESS / MSS) Standard Interface iView (me21n Transaction) Standard Interface Time Sheet Booking Business Content (ESS / MSS) BTR

ERP

EP

Create PO

Figure 11: The scenarios which are directly executed on the portal Scenario 3 - A business trip request (BTR) is created by users working in the portal by HTTP web recording of screens. The BTR is created in the backend ERP system through a standard interface. One user in RPT corresponds to one BTR in the ERP system. Scenario 4 Employees record their working time in time sheets via the enterprise portal. This is subsequently processed into time sheets in ERP. One user in RPT corresponds to one employee. Scenario 5 Users use the Enterprise Portal iView to create purchase orders manually, which then creates a PO in ERP. One RPT user corresponds to one user creating a PO.

Scenarios 3 & 4 are a part of ESS / MSS business content. RPT simulates employee log-in, recording weekly times, creating business trip requests and creating POs.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 18 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

4.4

BW Centric Scenario 6
RPT ERP EP BW

6 Portal Scripts

HTTP (Web Recording)

Triggering BW Report Screens

Standard Interface

Execute Standard BW reports

Figure 12: The BW centric scenarios. Scenario 6 RPT simulates users who log into EP to execute three URL iViews triggering standard BW reports on the PO transaction data. One RPT user corresponds to one user creating a report.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 19 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

5 Virtualization Measurements with Realistic Workload

The purpose of our tests is to prove the effective workload balancing of virtualized servers running an SAP application in a Linux (SLES 10) OS under changing workload conditions on IBM System x enterprise servers. For the virtualization load tests, the Linux guest OS monitoring is done with RPT from the RSTATD daemons inside the VMs and VMware vSphere across the whole enterprise system including the VMs. The administration of the VMs on vSphere happens according to standard guidelines. Links to further resources are in the appendix (Chapter 10.4). The following diagram shows one outcome of a planning session to determine how to distribute the different SAP applications across the available enterprise systems. Initially, we planned to run each 4 core system with 4 virtual CPUs, 8GB of memory and 512 MB of reservation per VM. This changed over the testing cycle, according to the ongoing monitoring of the resource consumption by the SAP application scenarios. A detailed configuration plan of the actual VM distribution is found in the appendix (Chapter 10.4).

Figure 13: The initial planning of system distribution and setup of the VMs

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 20 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

5.1

Virtualization Setup

Virtual Center (opt. SRM)

EP Prd passive (cluster)

Solution Manager SM1

EP Prd anvm57

ECC Dev R63

ECC Prd R61

SAP NW70

SAP NW71

ECC QA R62

SAP CRM

BW Prd BI7

SAP SRM

System x IBM 2 X3850 M2 16 cores 80 GB RAM

System x IBM 1 X3850 M2 16 cores 68 GB RAM

VMware 4.0 vSphere

VMware 4.0 vSphere

X86 / Windows RPT 2 Agent Controller 2 RPT 1 Workbench +AC1 TSM Server X86 Linux

SAP Systems used to simulate other load, not related to the Big X Business Process Scenario

FC SAN Switch Ethernet Switch

IBM XIV Storage System

Figure 14: Schematic System Layout

5.2 Throughput Tests on two x3950 M2 The six scenarios, as described in Chapter 4, simulated in RPT, cover 1000 users in total with a think time varying randomly between 2 and 10 sec between each step. This provided the necessary throughput to show a standard use case of the SAP application. The tests simulated the following distribution of users and orders. Scenario 1: PI Purchase Order creation through web services using 650 users over two iterations. This workload corresponds to approximately 4000 purchase orders with one material item per hour. Scenario 2: ERP Goods Receipt through 150 SAPGUI users with one iteration. This workload corresponds to approximately 450 goods receipts with one material item per hour. Scenario 3: EP Employee Self Service of a business trip request of 40 users with three iterations. This workload corresponds to approximately 350 business trip requests per hour. Scenario 4: EP Employee Self Service of time booking of 40 users with three iterations. This workload corresponds to approximately 350 time sheet updates per hour. Scenario 5: EP Purchase Order creation using 80 portal users with two iterations. This workload corresponds to approximately 500 purchase orders with one line item per hour. Scenario 6: EP Three different BW queries against the purchase orders triggered from Scenario 5 with 40 portal users using three sessions each with two iterations. This workload corresponds to approximately 700 queries in BW per hour.
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 21 of 120

Virtual Center

PI Prd PI7

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 15: Utilization of the VMs on the two x3950 M2 Figure 15 shows the percentage of idle CPU time over the testing period in the four SAP systems, ERP ECC (blue), BW (yellow), EP (red), and PI (green). Explanation: CPU utilization is comprised of the portions usr, system, idle and wait. The unit idle time is selected because it is the inverted value of the sum of usr, system and wait. That means, that if the line is at 100 (top end) there is no load and if the line is at zero the system is fully utilized with usr + system + wait. The whole simulation run took an elapsed time of about 18 minutes, as depicted via the red bracket in Figure 15. It is obvious from the four lines in the chart, that we have a good variation of workload in the various VMs in the two-system landscape. The detailed results of the throughput are available in the appendix chapter 10.2. 5.3 CPU Utilization The following two figures show the CPU utilization of our system named IBM1 and for the individual VMs contained within IBM1. VMware vCenter server samples the data point, in this case CPU utilization, every 20 seconds.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 22 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

CPU Utilization IBM1 CEC

100 90 80 70 Percent 60 50 40 30 20 10 0 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5:12:02 5:13:42 5:15:22 5:17:02 5:18:42 5:20:22 5:22:02 5:23:42 5:25:22 5:27:02 5:28:42 5:30:22 5:32:02 PM PM PM PM PM PM PM PM PM PM PM PM PM Time CPU utilization

Figure 16: CPU utilization IBM1 on system (CEC = Central Electronics Complex) level

CPU utilization IBM1 all VMs

250 Percentage added 200 150 100 50 0
01 27 0 5 /2 :1 2: 5/ 01 02 27 0 /2 5:1 P 5/ 010 3:4 M 27 2 /2 5:1 P 01 5: M 5/ 22 27 0 /2 5:1 P 5/ 010 7:0 M 27 2 /2 5:1 P 5/ 010 8:4 M 27 2 P /2 5:2 01 0: M 5/ 22 27 0 /2 5:2 P 5/ 010 2:0 M 27 2 /2 5:2 P 5/ 010 3:4 M 27 2 P /2 5:2 01 5: M 5/ 22 27 0 /2 5:2 P 5/ 010 7:0 M 27 2 /2 5:2 P 01 8: M 5/ 42 27 0 /2 5:3 P 01 0: M 2 0 5: 2 P 32 M :0 2 PM

SRM CRM R61 EP SolMan R63

5/

Figure 17: CPU utilization IBM1 all VMs added The next two charts show the CPU utilization on IBM2 and for the individual VMs contained within IBM2.

5/

27 /2

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 23 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

CPU Utilization IBM2 CEC

100 90 80 70 60 Percent 50 40 30 20 10 0 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5:12:02 5:13:42 5:15:22 5:17:02 5:18:42 5:20:22 5:22:02 5:23:42 5:25:22 5:27:02 5:28:42 5:30:22 5:32:02 PM PM PM PM PM PM PM PM PM PM PM PM PM Time CPU Utilization

Figure 18: CPU utilization IBM2 on system (CEC) level

CPU utilization IBM2 all VMs

250 Percentage added 200 NW71 150 100 50 0
01 27 0 5 :1 /2 5/ 010 2:0 2 27 PM /2 5 :1 5/ 010 3:4 2 27 P /2 5 :1 5: M 5/ 010 22 27 PM /2 5 :1 5/ 010 7:0 2 27 P /2 5 :1 01 8: M 5/ 27 0 5 42 PM :2 /2 5/ 010 0:2 2 27 PM /2 5 :2 5/ 010 2:0 2 27 P /2 5 :2 3: M 5/ 010 42 27 P /2 5 :2 5: M 01 5/ 27 0 5 22 PM /2 :2 5/ 010 7:0 2 27 P /2 5 :2 8: M 01 5/ 42 27 0 5 P :3 /2 0: M 01 22 0 5: 32 PM :0 2 PM

NW70 PI R62 BW

5/

Figure 19: CPU utilization IBM2 all VMs added For both machines, it is obvious that the peaks (actually troughs) of each VM occur at different times. If we had installed the landscape in a traditional fashion, we would have lots of unused capacity in off peak hours (so-called white space). In a virtualized setup, where we share the physical resources, the white spaces can be minimized. The silhouettes of the curves IBM SAP International Competence Center, Walldorf, Germany Page 24 of 120
2010 Global Alliance Solutions, SAP

5/

27 /2

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

of the individual VMs (Figure 17 & Figure 19) match nicely to that of the system views (Figure 16 & Figure 18), allowing peaks and troughs to even out. The efficiency gained by using virtualization is very dependent on the individual workload. In our case, we see that in IBM1 the overall physical system (Figure 16) shows a peak of 80 percent CPU utilization, whereas, in Figure 17, the same peak actually shows the sum of all VMs at a little over 200 percent in total using the same physical infrastructure. In IBM2 it is even more significant. The overall CPU utilization peaks at 70 percent (Figure 18) and the sum of the individual VMs is about 220 percent (Figure 19). We use the term CPU overcommitment for this behavior. In our tests with a realistic workload mix, we were able to overcommit our CPU resources to a factor between 2.5 and 3. This means that if one were to purchase separate servers for each VM, you would require three times the number of CPUs to cover the workload of the applications. When comparing the two system utilizations, there is potential for further consolidation. It is feasible to drive the utilization higher with more VMs running more applications. However, in a real customer scenario, longer-term monitoring of the CPU and memory utilization is heavily recommended to correctly determine if additional VMs and, therefore, additional work is possible. In conjunction with this monitoring, the distribution and size of the VMs can be improved iteratively. For each virtual machine using VMware ESX, one can specify the values of Shares, Reservation (minimum) and Limit (maximum) in vSphere. Shares determine a relative priority or importance of a VM. If a virtual machine has twice as many Shares for a resource as another virtual machine, it is entitled to consume twice as much of that resource. Shares are typically specified as high, normal, or low, using a 4:2:1 ratio. For SAP applications, it is recommended to set Shares to high for SAP production systems and normal or low for all other SAP systems. Also, it is recommended to check the box titled unlimited in the resource allocation configuration section of the VM in order to let the VM grow as necessary.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 25 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

5.4 Memory Utilization Overcommitment for memory in an SAP/VMware environment is not supported for production systems. We therefore assign the memory required to a fixed value for each VM. In the following Figures, the curves show the same total values between the system-wide consumption as well as the summed individual consumption. Test and QA systems could share the available memory, but this was not tested in our scenario. The first two charts show the memory consumption on IBM1 in total and the same for the individual VMs. Notice that the peak consumption of memory is around 12GB during the test cycle.
Memory Active IBM1 CEC
14000000

12000000

10000000

8000000 KB 6000000 4000000 2000000

0 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5:12:02 5:13:42 5:15:22 5:17:02 5:18:42 5:20:22 5:22:02 5:23:42 5:25:22 5:27:02 5:28:42 5:30:22 5:32:02 PM PM PM PM PM PM PM PM PM PM PM PM PM Time Memory Pages Active

Figure 20: Memory consumption IBM1 CEC level

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 26 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Memory Active IBM1 all VMs

14000000 12000000 10000000 8000000 6000000 4000000 2000000 0
/2 5/ 010 27 / 2 5: 1 2: 5/ 01 02 27 0 / 2 5: 1 P 5/ 010 3:4 M 27 2 / 2 5: 1 P 5: M 5/ 01 22 27 0 / 2 5: 1 P 5/ 010 7:0 M 27 2 / 2 5: 1 P 8: M 5/ 01 42 27 0 / 2 5: 2 P 5/ 010 0:2 M 27 2 / 2 5: 2 P 01 2: M 5/ 27 0 5 02 /2 P :2 3: M 5/ 01 42 27 0 / 2 5: 2 P 01 5: M 5/ 22 0 27 / 2 5: 2 P 7: M 5/ 01 02 27 0 / 2 5: 2 P 5/ 010 8:4 M 27 2 / 2 5: 3 P 01 0: M 22 0 5: 32 P M :0 2 PM

SRM CRM R61 EP SolMan R63

KB
5/ 27

Figure 21: Memory consumption IBM1 all VMs added The next two charts show the memory consumption for both IBM2 in total and for the individual VMs.
Memory Active IBM2 CEC
8000000

7000000

6000000

5000000

KB

4000000

3000000

2000000

1000000

0 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5/27/2010 5:12:02 5:13:42 5:15:22 5:17:02 5:18:42 5:20:22 5:22:02 5:23:42 5:25:22 5:27:02 5:28:42 5:30:22 5:32:02 PM PM PM PM PM PM PM PM PM PM PM PM PM Time Memory Pages active

Figure 22: Memory consumption IBM2 CEC level

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 27 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Memory Active IBM2 all VMs

8000000 7000000 6000000 KB 5000000 4000000 3000000 2000000 1000000 0
/2 5/ 010 27 / 2 5: 1 5/ 010 2:0 27 2 / 2 5: 1 P 01 3: M 5/ 42 27 0 / 2 5: 1 P 5/ 010 5:2 M 27 2 /2 5:1 P 5/ 010 7:0 M 27 2 / 2 5: 1 P 01 8: M 5/ 42 0 27 /2 5:2 P 5/ 010 0:2 M 27 2 / 2 5: 2 P 2: M 5/ 01 02 27 0 / 2 5: 2 P 01 3: M 5/ 42 0 27 / 2 5: 2 P 5/ 010 5:2 M 27 2 /2 5:2 P 01 7: M 5/ 27 0 5 02 /2 P :2 5/ 010 8:4 M 27 2 / 2 5: 3 P 01 0: M 22 0 5: 32 P M :0 2 PM

NW71 NW70 PI R62 BW

Figure 23: Memory consumption IBM2 all VMs added The memory is reserved for the individual VMs . Therefore the total consumption of memory (here approximately 7GB) is very similar overall in IBM2 (Figure 22) and when the individual VMs are added (Figure 23). Note: Please follow the published recommendations documented in the appropriate SAP notes along with the material provided in the References (9.1) regarding memory settings and infrastructure setup.

5/ 27

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 28 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

5.5 I/O Behavior on IBM XIV Storage We monitor various statistics of the storage subsystem directly using the IBM XIV GUI. Figure 24 shows the IOPS statistics per server. The red line represents the workload generated by IBM1 and the blue line represents IBM2. The maximum workload for both servers combined at any given time is about 3500 IOPS. The fields at the bottom of this Figure 24 show actual values (at the time when the screenshot was taken); they do not reflect the complete workload of the plotted area.

Figure 24: I/O statistics per server Note: The screenshots are from the XIV GUI during another simulation cycle. Therefore, the dates do not match the previous figures, yet the behavior shown is representative. Figure 25 shows cache hits for the storage system. The whole workload is being handled almost totally within the cache. If the previous figure is compared with this one, one sees that the lines almost match when the red and the blue lines are added together. This occurs because the biggest portion of the workload during the test is reading activity. Hence, the cache algorithm is able to prefetch or hold the required data in the cache.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 29 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 25: Cache Hit for all servers The following figure shows latency for the XIV storage system. Because of the high cache hit ratio the latency is negligible, as expected.

Figure 26: Latency in ms for the XIV storage system

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 30 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

5.6 CCMS Monitoring in Transaction os07n The SAP transaction os07n provides a view directly for the SAP Basis Administrator of the operation system utilization (Figure 27). Here are two example screenshots. The data is gathered from saposcol. Please find detailed documentation in SAP Note 1260719 SAPOSCOL: Detailed virtualization data.

Figure 27: SAP ERP Prod R61 os07n The next illustration is another example of os07n.Here the navigational bar on the top right is populated with all the SAP systems in the landscape (Figure 28).

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 31 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 28: Solution Manager os07n covering SAP ECC QA R62 SAP recommends setting up the monitoring functionality in the SAP Solution Manager. The systems in the SAP landscape will all have saposcol agents, which report the data into one singular data source. The SAP Solution Manager then provides the correlated data in one view. In the left navigation window, you can find the various monitored SIDs in the landscape and drill into the details. In addition, it is possible in transaction os07n to retrieve a history of the values over the lifecycle. It is possible to use IBM Tivoli Monitoring instead, but this was out of scope for this Proof of Concept.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 32 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

5.7 Throughput Tests on one x3850 X5 In this test we show the same workload (scenario 1 to 6, 1000 simulated users) running on one x3850 X5 system.

Figure 29: Utilization in the VMs on the x3850 X5 Figure 29 shows the percentage of idle CPU time over the testing period in the four SAP systems, ERP ECC (blue), BW (yellow), EP (red), and PI (green). The detailed results of the throughput are available in the appendix chapter 10.3. 5.7.1 CPU Utilization The following Figure 30 shows the CPU utilization overall. Similar to the tests on the two x3950 systems (IBM1 and IBM2), we see a utilization in total of maximum 40 percent on CEC level (Figure 30), but if we look at the values per VM added together as in Figure 31 we reach 200 percent, which would relate to an excellent overcommitment factor of 5.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 33 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

CPU Utilization X5 CEC

45 40 35 30 25 20 15 10 5 0 7/28/2010 7/28/2010 7/28/2010 10:37:31 10:35:51 10:34:11 AM AM AM

Percent

7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 10:47:31 10:45:51 10:44:11 10:42:31 10:40:51 10:39:11 AM AM AM AM AM AM Time Usage

7/28/2010 7/28/2010 10:50:51 10:49:11 AM AM

Figure 30: CPU utilization X5 on system (CEC) level Here are the individual CPU utilizations of the VMs stacked on top of each other. (Figure 31)
CPU utilization all VMs x3850 X5
250 BW 200 Percentage added CRM EP NW 70 150 NW 71 PI 100 ECC Prd R61 ECC QA R62 ECC Dev R63 50 SolMan SRM 0
7/ 28 /2 01 0 10 7/ :3 28 4: /2 11 01 AM 0 10 7/ :3 28 5: /2 51 01 AM 0 10 7/ :3 28 7: /2 31 01 AM 0 10 7/ :3 28 9: /2 11 01 AM 0 10 7/ :4 28 0: /2 51 01 AM 0 10 7/ :4 28 2: /2 31 01 AM 0 10 7/ :4 28 4: /2 11 01 AM 0 10 7/ :4 28 5: /2 51 01 AM 0 10 7/ :4 28 7: /2 31 01 AM 0 10 7/ :4 28 9: /2 11 01 AM 0 10 7/ :5 28 0: /2 51 01 AM 0 10 :5 2: 31 AM

Time

Figure 31: CPU utilization X5 all VMs added

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 34 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

5.7.2 Memory Utilization The following two illustrations show the active memory used by the test on X5.
Memory Active X5 CEC
20000000 18000000 16000000 14000000 12000000 KB 10000000 8000000 6000000 4000000 2000000 0 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 7/28/2010 10:34:11 10:35:51 10:37:31 10:39:11 10:40:51 10:42:31 10:44:11 10:45:51 10:47:31 10:49:11 10:50:51 AM AM AM AM AM AM AM AM AM AM AM Time Active

Figure 32: Memory active in total on X5

Memory Active X5 all VMs
20000000 18000000 16000000 14000000 12000000 KB 10000000 8000000 6000000 4000000 2000000 0
10 :3 7/ 4: 28 11 /2 01 AM 0 10 :3 7/ 5: 28 51 /2 01 AM 0 10 :3 7/ 7: 28 31 /2 01 AM 0 10 :3 7/ 9: 28 11 /2 01 AM 0 10 :4 7/ 0: 28 51 /2 01 AM 0 10 :4 7/ 2: 28 31 /2 01 AM 0 10 :4 7/ 4: 28 11 /2 01 AM 0 10 :4 7/ 5: 28 51 /2 01 AM 0 10 :4 7/ 7: 28 31 /2 01 AM 0 10 :4 7/ 9: 28 11 /2 01 AM 0 10 :5 7/ 0: 28 51 /2 01 AM 0 10 :5 2: 31 AM

BW CRM EP NW 70 NW 71 PI ECC Prd R61 ECC QA R62 ECC Dev R63 SolMan SRM

7/ 28 /2 01 0

Time

Figure 33: Memory active on all VMs on the X5

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 35 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Again, due to the fixed assignment (reservation) of memory for the individual VMs the total line (Figure 32) and the added individual data from the VMs (Figure 33) look almost the same. For this test we did not pull I/O performance data from the XIV. It was comparable with the data we saw from the test with the two x3950 systems.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 36 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

6 Business Resiliency
Business resiliency describes how to maintain continuous business operations; it comprises these six disciplines, as specified by IBM: Integrated risk management - Identify risks to your business operations and utilize technology to understand, respond to, and manage those risks Regulatory compliance - Comply with current government and industry regulations and standards, especially regarding the integrity and availability of information Security, privacy and data protection - Ensure the security and privacy of data, information, systems and people with the right policies, methods, tools and overall governance Knowledge, expertise and skills - Address the resilience of your business by assuring that you have the right resources in the right place at the right time Market readiness - Enhance your ability to sense and respond to shifting customer demands and new market opportunities Continuity of business operations - Maintain business operations in the event of an outage with processes and infrastructures that are responsive, highly available and scalable This paper focus on Continuity of business operations, how to manage, and how to reduce the planned and unplanned outage (downtime) of SAP systems. Planned outage A planned outage is, as the name suggests, an advance, planned downtime of an application, in this case SAP. Typically this is done for release upgrade, installing a fix package (SAP, OS, firmware, ), or if hardware needs to be replaced. Unplanned outage An unplanned outage occurs in an even of failure, a hardware component breaks down, or software crashes. Both scenarios cannot be avoided, but with the right techniques, the impact of the downtime and the duration of a downtime, can be reduced or mitigated. Mitigation for Planned Outage An SAP or OS level upgrade, as well the installation of most SAP and OS fixes and patches, requires a shutdown and restart of SAP, or even a reboot of the OS. This behavior is independent of the virtualization techniques deployed, and independent of VMware, and is therefore not discussed in this paper. The replacement of hardware, or adding new hardware, can make use of virtualization in order to prevent a shutdown or restart. See section 6.5. Preventing or mitigation for Unplanned Outage Protection against infrastructure failures High Availability (HA) handles infrastructure issues, such as broken hardware or software crashes. It is possible to recover from a single component failure with a backup of this component.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 37 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

At the server level this can be achieved with a secondary server in combination with cluster management software. VMotion can be used in order to achieve high availability of the infrastructure. Protection against site failures or data corruption Disaster Recovery (DR) extends the HA scenario in order to provide protection if more than one component fails, up to a site failure in case of power loss or flooding. Also, DR can handle logical errors, such as data corruption or data loss with the SAP data base, by providing the capability to restore the data to an earlier point in time. 6.1 Concepts evaluated in this Proof of Concept - Standard file backup with the TSM BA-Client This method provides the restore capability of a single or multiple files (e.g. configuration files) that have been deleted accidentally or mis-edited. Online Backup of the SAP data base server see section 10.7.1 FlashCopy Backup of the SAP data base server see section 10.7.2 Both methods protect against a data loss or data corruption of the DB server. VMware Consolidated Backup to backup the entire VM system This method protects again the loss of the entire OS operation system, including all installed applications, in case of a RAID, LUN, or file system error. VMware Consolidated Backup is not part of this POC. VMware Site Recovery Manager (SRM) to protect against a site failure see section 6.2

Backup/Restore is the basis for a DR. This sample procedure outlines a simple backup plan for the discussed SAP landscape: 1. Initially install and configure the operating system and applications 2. Shutdown each VM so that they are offline 3. Create an initial snapshot of all volumes by using IBM XIV GUI. 4. Start the VMs and the applications to be used in a production environment 5. Setup a schedule (TSM Server) to backup all standard files, like executable and configuration files, on a daily basis 6. Setup a schedule (TSM Server) to backup the DATA and LOG files of the SAP data base sever by using FlashCopy technology (FlashCopy Manager), on an hourly basis.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 38 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

6.2 Integrated DB2 backup with TSM for ERP TSM for ERP protects SAP system data and integrates with the database-specific utilities of IBM DB2. Data Protection for SAP improves the availability of SAP database servers and reduces administration workload with automated data protection features designed specifically for SAP environments. Figure 34 shows the capability of IBM Tivoli Storage Manager Suite in order to backup SAP systems: The online DB DATA and LOG files, the SAP and data base (DB2) application files, as well as the Linux OS image.

Figure 34: IBM Tivoli Storage Manager capabilities TSM for ERP focuses on database objects to backup and restore database contents and database specific control files, for example, the database configuration, the history and the log file header, and offline DB2 log files. Other files, such as SAP and DB2 executable, can be backed up using the TSM BackupArchive Client. This is important for disaster recovery purposes, as all SAP and DB2 executable files must be available before using Data Protection for SAP to restore and recover the database. Another option is to use VMware Consolidated Backup to restore the OS image, including the application executable. For a detailed description of how to setup and use TSM for SAP, please see chapter 10.7.1 A: Data protection with Tivoli Storage Manager for ERP.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 39 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

6.3 Data protection with IBM FlashCopy Manager (FCM) The IBM FlashCopy Manager provides the functionality to perform a backup based on FlashCopy. With Version 2.2 FCM supports the LINUX x86_64 operation system as well as the IBM XIV storage system. How to use FCM for this environment is documented in this whitepaper, including backup, restore, and cloning: http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101703 This section provides additional information on how to use FCM in combination with VMware. The discovery of flash copied LUNs and how to access them from LINUX as VM is different comparing if LINUX runs on the server natively. The only process derivation from the standard process during the FlashCopy based backup of SAP within VMware is the mount operation of the FlashCopy volumes by the backup (proxy) node. Major steps to perform a FlashCopy backup: 1. 2. 3. 4. 5. 6. 7. Prepare LINUX systems Create FlashCopy Backup (DB2 tool) Identify the new FlashCopy LUNs on XIV (LINUX tools) Map the FlashCopy LUNs to ESX Server (XIV CLI) Provide LUN access from ESX server to LINUX (VMware GUI or CLI) Scan SCSI bus on LINUX (LINUX tools) Run FCM in order to process the FlashCopy backup

Steps 3 to 6 are in addition to the standard procedure in order to adapt to a VMware environment. The detailed description of these steps is documented in chapter: 10.7.2 A: Data protection with IBM FlashCopy Manager (FCM).

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 40 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

6.4

VMware Site Recovery Manager

6.4.1 What is VMware Site Recovery Manager? VMware Site Recovery Manager (SRM) is a disaster recovery management solution for VMware infrastructures. SRM helps accelerate recovery by automating the recovery process and by simplifying the management of disaster recovery plans through automated integration. The solution provides reliable recovery by eliminating complex manual recovery steps and helps to eliminate disruptive testing of recovery plans. SRM integrates tightly with VMware Infrastructure, VMware vCenter, and storage replication software from IBM to process site failovers of rapid, reliable, and affordable recoveries. It eases disaster recovery risk, as well as protects all critical systems and applications. 6.4.2 Summary of VMware SRM Key Features As IT environments get more and more complex, organizations find it increasingly difficult to provide disaster recovery solutions that meet their needs. SRM helps organizations address the challenges of traditional disaster recovery. SRM manages the failover from production data centers to disaster recovery sites. If a datacenter is distributed across two sites, SRM enables the failover between each site. SRM also assists in planning failovers, for example, in scenarios such as data center migrations, by automating and simplifying the process of failing over to the new data center. Manage disaster recovery plans SRM allows users to create, update, and document recovery plans as an integrated part of managing their VMware virtual environments Steps and resources that make up the failover process are managed directly from VMware vCenter No manual runbook documentation and maintenance of the steps and resources required for failover is necessary. Non disruptive testing SRM uses storage snapshot capabilities to perform recovery tests without losing replicated data Connects virtual machines to an existing isolated network for testing purposes Automates execution of recovery plans Customizes execution of recovery plans for testing scenarios Automates cleanup of testing environments after completing failover tests. Automated failover SRM initiates recovery plan execution from VirtualCenter with a single button Automates promotion of replicated datastores for use in recovery scenarios with adapters created by leading storage vendors for their replication platforms Runs user-defined scripts and halts during recovery Reconfigures VMs IP addresses to match network configuration at failover sites Manages and monitors execution of recovery plans within VirtualCenter.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 41 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

6.4.3 Overview of the system environment The following Figure 35 provides a graphical overview of the system environment for SRM tests in this Proof of Concept.

Figure 35: Graphical overview of the system environment We used the following steps to implement a DR scenario using SRM: 1. At the primary site, VMware ESX is installed on an IBM System x3950 X5 server on the local hard disk. 2. The VMware vCenter including the SRM plug-in runs in a VM (Microsoft Windows 2003) and the data resides on the XIV storage system. 3. The storage replication adapter (SRA) is installed in the same Microsoft Windows virtual machine. 4. The SAP systems are installed in separate virtual machines as described in the Appendix (10.4). 5. At the secondary site (also called recovery or remote site), VMware ESX is installed on an IBM BladeCenter HS22 server on the local hard disk. We chose a different server for the secondary site to simulate a customer hardware upgrade as well as to show the interoperability of the software stack on IBM System x servers. 6. The two VMware vCenter servers are connected through the SRM plug-in. A synchronous mirror relationship between the storage systems is set up to ensure a
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 42 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

consistent database state. On the secondary storage site, the mirrored storage volume is attached to the host; however, it is not added to the storage pool in VMware vCenter. The result of these configurations on the secondary site creates a read-only mirrored storage volume and an inactive virtual machine (as a mirror of the original machine), which cannot be started because it has no active storage connected. Note: The storage systems must be connected through a SAN (for fibre channel connection) or Ethernet (for iSCSI connection) switch. Testing of the Scenario To demonstrate that the solution works in SAP environments, the following scenarios were defined and run: 1. Simulate a disaster using SRM, while not changing the infrastructure. 2. Perform an actual failover by starting a job in SAP and shutting down the VMware vCenter server on the primary site. Analyze the behavior while performing the failover. 3. Reverse step 2 and shut down the VMware vCenter server on the secondary site and perform a failback. The scenarios and test results are described in more detail in the following sections.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 43 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

6.4.3.1

Running the SRM Test

To verify that a DR plan is working, run a test that simulates the steps carried out during a disaster. The Figure 36 below shows the steps as planned to be performed during a disaster using SRM. It also shows that all steps have been completed successfully after such a test.

Figure 36: Steps of the recovery plan During the planned disaster test, the environment looks as follows: Primary Site All user sessions remain online, and all jobs in SAP continue to run. Secondary Site A snapshot is taken from the mirrored volume on the XIV storage. It is unlocked, made readable, and assigned to the VMware vSphere host. On the ESX host the snapshot is added as a new storage volume and assigned to the protected virtual machines. The low-priority virtual machines are suspended. The protected dummy virtual machines are started. Figure 37 shows the snapshots taken on the IBM XIV for the SRM test. The snapshots having names starting with srm_testfailover_ represent the snapshots created for the failover test.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 44 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 37: Snapshots created for the SRM failover test When the test steps are completed, a check is done to verify that the protected virtual machines are started correctly. After successful verification click Continue in the vCenter message box to bring the environment back to its original state. Note: Be careful to start one or more SAP systems in a protected virtual machine. The SAP systems might interconnect with the original SAP landscape that is still online! It can be seen that SRM is able to manage the customization of the guest network on the secondary site, for example, during a failover test. There is an option to bring virtual machines up in a test bubble or into the secondary network. More detail on this procedure can be found at: http://blogs.vmware.com/uptime/2009/01/how-to-exploit-the-test-bubble-for-all-itsworth.html and http://viops.vmware.com/home/docs/DOC-1449
6.4.3.2 Running a Failover

The following section describes how to run a failover. To simulate the failover, the VMware ESX host on the primary site was powered off. In either case, the administrator only needs to press a button in the SRM on the secondary site in order to trigger the recovery process for all virtual machines that are protected within the selected recovery plan. SRM then performs the following steps: 1. Shut down the protected virtual machines on the primary site (if they are still running).
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 45 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

2. The Storage Replication Adapter (SRA) inactivates the mirror relationship of the storage devices and switches the mirrored volumes from slave to master, so that it becomes writeable. 3. The volumes of the secondary storage device are assigned to the protected dummy virtual machine on the secondary site, so that the protected virtual machine becomes a real virtual machine. 4. If running, any low-priority virtual machines on the secondary site are suspended. 5. The new real virtual machine is started. We use a simulation of scenario 1 (purchase orders from PI to ERP): 1 user with 500 iterations, which equals to 500 purchase orders to be processed. The transaction WE05 in ECC was done before the failover was triggered. In Figure 38, a check in ERP transaction shows the IDOCS processed in SAP system R61 without disruption.

Figure 38: Screenshot of we05 in ECC (R61) We started the scenario 1 again with 1 user and 500 iterations. After one minute, we powered off the primary site VMware vSphere host and started the failover on the secondary site. The RPT test log in Figure 39 showed that 283 purchase orders were sent to PI (Figure 40).

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 46 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 39: RPT Test log

Figure 40: processed XML messages from PI Figure 41 shows the recovery plan during execution. The first recovery step shows an error because the virtual machines on the primary site could not be shut down as the host was powered off.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 47 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 41: Running the recovery plan on the secondary site Figure 42 shows the IBM XIV mirroring relationship being inactive; the mirrored volumes became masters.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 48 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 42: XIV configuration done by the storage replication adapter triggered by running the recovery plan When the SAP systems were running again on the secondary site, we checked in transaction sxmb_moni in PI and found out that 251 messages were processed successfully by PI (meaning sent to ERP successfully as in Figure 43) and 2 failed. This means that 253 messages arrived in the PI integration engine.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 49 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 43: Transaction we05 in ECC R61 From 283 requests sent by RPT, only 253 arrived in the PI integration engine. The remaining 30 IDOCs were still hanging in the queues. After the check, we started another run again with 1 user with the same 500 iterations (same 500 purchase order numbers). The result was that the 251 IDOCS which were already sent successfully to ERP in the previous run got a red flag meaning IDOC already exists (status 51). The remaining 249 IDOCs got a green flag which means the IDOCs were processed successfully (status 53). The 30 IDOCs, which were still hanging in the PI system queue from the previous run, arrived in ERP as soon as the PI system started processing after the failover. Those 30 IDOCs also had a red flag (status 51) after being sent to the ERP system, because all of the 500 IDOCs were processed already. This results in 281 failed IDOCs in ERP. Transaction WE05 showed that in total 530 IDOCs (249 + 281) were sent to ERP (Figure 44). This proves that no purchase order was lost due to the failover.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 50 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 44: Screenshot we05 in ECC after failover completion

6.4.3.3 Running a Failback

In case of a failback, typically only the primary and secondary sites need to be switched in SRM and on the XIV, and the recovery plan needs to be run on the former primary site. Because the primary site was only powered off but not destroyed during the failover, the failback could be processed easily. We performed the following steps manually to complete a failback: 1. On the former primary site, we removed the recovered virtual machines. 2. On the former primary site, we cleaned the protection setup. 3. On the former secondary site, we removed the recovery plan. 4. On the XIV, we switched the roles of the mirrored volumes on the primary site from master to slave, and we manually activated the synchronization on the secondary site. 5. We configured the former secondary site as the new primary site. Specifically, we configured the connection, array managers, inventory mappings, and the protection groups of the new primary site. 6. On the former primary site, we created the recovery plan. 7. On the former primary site, we ran the recovery plan. Note: The former primary site storage subsystem is automatically defined as the secondary site storage subsystem and the former secondary site storage subsystem is automatically defined as the primary site storage subsystem. 8. After the failback was processed, we switched the roles of the mirrored volumes on the secondary site back from master to slave, and we manually activated the synchronization on the primary site.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 51 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

The environment was back at its initial state and ready to be used to run a failover again, if desired. 6.5 How to Reduce the Time for Planned Outages The standard procedure to move application systems seamless without loss of service is to use VMware VMotion. We use VMotion to test reducing a planned failover outage time using the previous scenario with the SAP system R61. While running an RPT simulation (approx. 1000 users actively working according to the description in Chapter 4), VMotion allows all users to continue working without any loss of work. The vCenter of vSphere delivers a protocol about the live migration of the VM: anvm62_ECC_R61 VC-BIGX Start 5/26/2010 11:55:39 AM End 5/26/2010 12:01:07 PM Completed VMotion is also the base for VMware Dynamic Resource Scheduling (DRS). VMware DRS continuously balances computing capacity in resource pools to deliver the performance, scalability and availability not possible with physical infrastructure. VMware DRS allows you to: Improve service levels for all applications. VMware DRS continuously balance capacity will ensure that each virtual machine has access to appropriate resources at any point in time. Easily deploy new capacity. VMware DRS will seamlessly take advantage of the additional capacity of new servers added to a resource pool by redistributing virtual machines without system disruption. Automate planned server maintenance. VMware DRS can automatically migrate all virtual machines off physical servers to enable scheduled server maintenance with zero downtime. This functionality was not tested in PoC.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 52 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

7 Cloning
From a business perspective, cloning requires a 1:1 copy of an existing system. On the infrastructure layer, it is possible to create the copy without consuming additional space on the storage system. With XIV technology, we are able to perform this using snapshot functionality. Possible use cases for SAP are copying test systems for an education landscape or performing a DB system refresh from PRD to QA. There are two cloning methods: Cloning of one SAP system with a SID change (ABAP and some Java components) through standard SAP procedures. Cloning of one SAP system without a SID change and the newly cloned system is isolated using network methods, such as using VLANs. For this method, the network switch needs to be capable of running VLANs. SAP system cloning also known as homogeneous system copy (HSC) creates a copy of an SAP system where the operating system and the database version are the same on the target system as on the source system. Typically this data replication is performed thorough a redirected restore. In addition, the preand post-processing need to be executed on the production database. Both steps are timeconsuming. By using the IBM SAP cloning solution the pre- and post-processing on the production sever will be eliminated, and the data replication is performed through storagebased copy functions, called snapshots on the IBM XIV storage system. By using snapshots (storage-based copy functionality), the impact to the production system is minimal during the cloning process; also the storage consumption is minimized, because snapshots are working space efficiently. Only changed or new blocks require additional space, and a full physical full copy is not required. In order to execute the IBM SAP cloning solution efficiently and correctly, we recommend an appropriate naming convention to identify the volumes (LUNs). An example is available in the appendix (chapter 10.5). The screenshots in chapter 10.8 describe best how to clone an SAP system using the IBM SAP cloning solution, step by step.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 53 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

8 IBM DB2
8.1 Storage Layout: We use the SAP standard directory sapdata1 to sapdata4 layout. These directories are placed in one single filesystem using a single LUN as the underlying storage. It is attached to the server via a FC SAN switch where the data is stored and distributed on the IBM XIV storage system. The automatic IBM XIV data distribution algorithm guarantees that all SAP data i.e. data files (SAPDATA) and transaction logs (SAPLOG) are equally distributed over all available disks. No more sophisticated planning of LUN layouts is necessary. Recommendation: Separate database logs and data files, and provide a separate volume or set of volumes for the local DB2 database directory. For more information: Best Practices Guide for SAP Applications on an IBM XIV Storage System http://www03.ibm.com/systems/resources/storage_solutions_pdf_xiv_sap.pdf?CACHEID=4b3022004e4 54ceb91c6f3901a2417e7 8.2 Concurrent I/O Parameter Settings In our scenarios we use SLES 10 SP2 and the EXT3 filesystem type, which supports direct I/O (for support matrix see the table below). In addition, some file systems such as IBM JFS2 or Symantec VERITAS VxFS also support enhanced Direct I/O (DIO), that is, the higherperforming Concurrent I/O (CIO). Since DB2 version 9.5, DB2 supports DIO and CIO with the NO FILE SYSTEM CACHING table space clause by default. When this feature is set, the database manager automatically takes advantage of DIO on file systems and avoids double caching by bypassing the filesystem cache. This new default provides an increase of throughput over buffered I/O on heavy transaction processing workloads and rollbacks. CIO also reduces INODE contention on database files, which can become a bottleneck for very large database files. For a reference, see SAP note 1353421 DB6: How to deactivate file system caching in V9.5.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 54 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

With the command: db2 get snapshot for tablespaces on <SID>, you can check the setting of the parameter: FILE SYSTEM CACHING. It should be set to NO, the default setting for new installations. Platforms File system type and minimum level required DIO or CIO requests submitted by the database manager when NO FILE SYSTEM CACHING is specified DIO Default behavior when neither NO FILE SYSTEM CACHING nor FILE SYSTEM CACHNG is specified NO FILE SYSTEM CACHING

Linux distributions SLES 10 SP2 or higher, and RHEL 5.2 or higher (in these architectures: x86, x64, POWER) Linux distributions SLES 10 SP2 or higher, and RHEL 5.2 or higher (in these architectures: x86, x64, POWER)

ext2, ext3, reiserfs

VERITAS Storage Foundation for DB2 4.1 (VxFS) VERITAS Storage Foundation for DB2 5.0 (VxFS)

CIO

NO FILE SYSTEM CACHING

Table 1: Operation system, file system type and caching settings 8.3 Installation Parameters During the standard SAP installation with DB2 for LUW on Linux, the following features either are switched on or are activated by default. For this Proof of Concept we used the automatic settings of DB2 applied during the SAP installation process. 8.3.1 Instance Memory The instance memory is set to a fixed value (Figure 45). For the R61 and PI7 systems we have assigned a lower percentage of roughly 16 percent or 2.6GB of 16GB total RAM as the instance memory. This is possible since we have not observed any memory bottleneck during our tests. A better rule of thumb for production systems is: Central ABAP-System (DB and App-Server on same machine) ~30 percent of real memory for database shared memory and sort memory. Distributed ABAP-System (Database on dedicated machine) ~60 percent of real memory for database shared memory and sort memory.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 55 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 45: SAP Installation Process DB2 Instance Memory 8.3.2 Compression and Virtual Tables If DB2 compression is switched on during installation, then all tables and, since DB2 9.7, all indices, will be compressed. In our scenarios we wanted to compress only the largest tables/indices, and therefore, we omitted this setting and activated compression directly for those specific tables with the SAP report /ISIS/ZCOMP. SAP Notes: 980067 (DB6: Using DB2 9 Row Compression) and 1378499 (DB6: CLI error CLI0112E with "REORGCHK_ALL" job) needed to be applied. It is also recommended to use deferred table creation, also referred to as virtual tables (Figure 46). DB2 allocates space in the tablespaces for empty tables, and SAP systems can contain several 10,000s of empty tables, which means that more than 10GB of empty space may be used. In addition, empty tables have a negative effect on the system resources for DB2 autonomic functions, e.g. AutoRUNSTATS and AutoREORG, as they use memory in different areas (DBHEAP, and so on). To save resources, the SAP software replaces empty tables with special views (virtual tables).

Figure 46: SAP Installation process compression, deferred table creation screen
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 56 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

8.3.3 Automatic Storage Automatic storage was introduced in DB2 V 8.2.2, and since DB2 9 the database and the SYSCAT tablespace are created with the automatic storage clause. For the SAP tablespaces the user can choose between DMS tablespace in autoresize mode or automatic storage tablespaces (AutoStorage) by activating the checkbox Use Automatic Storage Management for Tablespaces (AutoStorage) (Figure 47). With DB2 9.7 it is also possible to manually reclaim unused space. In this project we have activated automatic storage also for SAP tablespaces, thus the storage was managed at database and not at tablespace level. The responsibility of creating, extending and adding containers was taken over by the database manager. (Figure 47) More information can be found on SDN: Enable Automatic Storage for Your SAP Database and Table Spaces: http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/70da5b8c-d651-2c1037ac-9eb76853d987?QuickLink=index&overridelayout=true

Figure 47: SAP Installation Process Automatic Storage, Sapdata directories The detailed configuration files exemplary for R61 (ECC Prd) are available in the appendix (10.9).

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 57 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

8.4 SAP DBA Cockpit for DB2 LUW The DB2 LUW administration functions are fully integrated into the SAP application, and are provided via the SAP DBA Cockpit (Figure 48). We use the SAP DBA Cockpit during the PoC to carry out administration tasks such as database reorganizations, backups, activating table compression, monitoring and performance analysis. (For more information see IBM EBook SAP DBA Cockpit - Flight Plans for DB2 LUW, http://bit.ly/DB2_SAP_PDF)

Figure 48: Example of an SAP DBA Cockpit for DB2

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 58 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

9 References
9.1 VMware Virtualization of SAP Applications with VMware vSphere on IBM System x3850 X5 http://www.ibm.com/common/ssi/fcgibin/ssialias?infotype=SA&subtype=WH&appname=STGE_XS_IN_USEN&htmlfid=XSW03 023USEN&attachment=XSW03023USEN.PDF Virtualization of SAP Applications with VMware ESX Server on IBM System x3850/3950 M2 http://www.vmworld.com/servlet/JiveServlet/downloadBody/3700-102-14746/Virtualisation%20for%20SAP%20with%20ESX%20on%20IBM%20System%20x3950V1-1.pdf http://www.ibm.com/common/ssi/fcgibin/ssialias?infotype=SA&subtype=WH&appname=STGE_XS_IN_USEN&htmlfid=XSW03 023USEN&attachment=XSW03023USEN.PDF Virtualize, consolidate and scale SAP with IBM System x and eX5 https://www-304.ibm.com/partnerworld/wps/servlet/mem/ContentHandler/XSS03017USEN Virtualization for SAP_on_IBM BladeCenter Solutions http://www.ibm.com/partnerworld/pwhome.nsf/vAssetsLookup/virtualisation_for_sap_on_ib m_bladecenter_v1.5.pdf/$File/virtualisation_for_sap_on_ibm_bladecenter_v1.5.pdf vSphere Basic System Administration http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_admin_guide.pdf VMware Consolidated Backup integration with IBM Tivoli Storage Manager http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105269 To provide a predefined starting point for a new SAP system, it is possible to work with templates in VMware. http://www.vmware.com/resources/techresources/408 Virtualization on Windows http://www.sdn.sap.com/irj/sdn/windows-virtualization Note 1409608 - Virtualization on Windows VMware Site Recovery Manager Further information on VMware SRM: www.vmware.com/products/srm/ Whitepapers related to VMware SRM, IBM storage and SAP landscapes: www.ibm.com/systems/storage/solutions/isv/#sap 9.2 DB2 IBM DB2 for Linux, Unix and Windows
http://www-01.ibm.com/software/data/db2/9/

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 59 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

IBM DB2 for Linux, Unix and Windows on IBM Developer Works http://www.ibm.com/developerworks/data/products/db2luw/ IBM DB2 V9.7 Information Center http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp The Self-Tuning Memory Manager (STMM): A Technical White Paper ftp://ftp.software.ibm.com/software/data/pubs/papers/stmm.pdf IBM E-Book SAP DBA Cockpit Flight Plans for DB2 LUW Administrator http://bit.ly/DB2_SAP_PDF Article on SAP SDN: The Evolution of the Memory Model in IBM DB2 for Linux, UNIX and Windows by Johannes Heinrich http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/b0aabcc9-afc1-2a10-5091b5cda33036b0 SAP on DB2 for Linux, UNIX and Windows in the SAP Developer Network (SDN) http://www.sdn.sap.com/irj/sdn/db6;jsessionid=(J2EE3417300)ID1717351450DB128124559 14985883617End SAP Note 1351160 - DB6: Using DB2 Version 9.7 with SAP Software SAP Note: 1329179 - DB6: DB2 V9.7 Standard Parameter Settings. 9.3 TSM TSM Server, Client, and TSM for ERP administration, installation, and user guides: http://publib.boulder.ibm.com/infocenter/tsminfo/v6r2/index.jsp Quick Start Guide to FlashCopy Manager for SAP on IBM DB2 Database with IBM XIV Storage System http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101703 Tivoli FlashCopy Manager information http://www-01.ibm.com/software/tivoli/products/storage-flashcopy-mgr/ FCM V2.2 requirements checklist http://www-01.ibm.com/support/docview.wss?&uid=swg21431385 FCM V2.2 installation and user guide http://www-01.ibm.com/support/docview.wss?rs=0&uid=swg24026421 Redbooks: SAP Backup using Tivoli Storage Manager http://www.redbooks.ibm.com/abstracts/sg247686.html?Open IBM Tivoli Storage Manager Implementation Guide http://www.redbooks.ibm.com/abstracts/sg245416.html?Open IBM Tivoli Storage Management Concepts http://www.redbooks.ibm.com/abstracts/sg244877.html?Open

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 60 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10 Appendix (A)
The following chapters contain additional data and screenshots, referred to in the description chapters above. The figures are mostly not explained in detail, and serve as backup information only. 10.1 A: Rational Performance Tester (RPT) Setup 10.1.1 RPT Configuration ESS Scenarios: https://sapneth3.wdf.sap.corp/trainingcontent Page 127 & 130 Overview Page 296 & 297 Specifics / Peak Loads

Rational Performance Tester (RPT) Version and installed features:

Figure 49
RPT system properties:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 61 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 50

RPT Agents:

Figure 51

Load Settings
User Load:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 62 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 52

Think Time:

Figure 53

Resource monitoring:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 63 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 54
Statistics:

Figure 55

Test Log:

Figure 56
RAM Usage:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 64 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 57

10.1.2 Scenario 2 Goods Receipt

Go to Goods Receipt Transaction: MIGO

Figure 58
Enter External PO Number and Press Enter:

Figure 59
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 65 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 60

Select Item OK Check box and Save:

Figure 61

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 66 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 62
Logoff

10.1.3 Scenario 3 Create Business Trip Request As a preparation for scenario 3 and 4 it is necessary to hire employees:
Log On Page:

Figure 63

Travel and Expenses Tab:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 67 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 64
Press Create travel request:

Figure 65

Enter following data and press Review:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 68 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 66

Figure 67
Save and Log off session:

Figure 68 10.1.4 Scenario 4 Working Time

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 69 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Log On Page:

Figure 69

Go to working time Tab:

Figure 70
Press Record working time:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 70 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 71

Select working period and press GO:

Figure 72
Maintain working hours and press Review:

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 71 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 73
Save and Log off session:

Figure 74

10.1.5 Scenario 6 BW Reports

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 72 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Log On Page:

Figure 75

Bi iViews Tab:

Figure 76

Press one of the report links: (In the following example SAMPLE2 is selected)

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 73 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 77
Log off

10.2 A: Rational Performance Tester Result Screens on two x3950 M2 10.2.1 Run Summary Executed Test SC_1_2_3_4_5_6 Active Users 0 Completed Users 1,000 Total Users 1,000 Elapsed Time [H:M:S] 0:17:52 Run Status Complete Displaying Results for Computer: All Hosts Page Summary Percent Page VPs Passed [for Run] 0 Total Page VPs Failed [for Run] 3 Total Page Attempts [for Run] 5,640 Total Page Hits [for Run] 5,637 Average Response Time For All Pages [ms] [for Run] 10,811 Response Time Standard Deviation For All Pages [for Run] 26,167.2 Maximum Response Time For All Pages [ms] [for Run] 219,047 Minimum Response Time For All Pages [ms] [for Run] 0 Transaction Summary Average Elapsed Time For All Transactions [ms] [for Run] Maximum Elapsed Time For All Transactions [ms] [for Run] Minimum Elapsed Time For All Transactions [ms] [for Run] Total Transactions Completed [for Run] 450 Total Transactions Started [for Run] 450 Elapsed Time Standard Deviation For All Transactions [for Run]

72,376 189,890 12,609

48,274.9

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 74 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.2.2 SAP Summary Executed Test SC_1_2_3_4_5_6 Active Users 0 Completed Users 1,000 Total Users 1,000 Elapsed Time [H:M:S] 0:17:52 Run Status Complete Displaying Results for Computer: All Hosts Total SAP Connection Errors [for Run] 3 Transaction Summary Average Elapsed Time For All Transactions [ms] [for Run] Maximum Elapsed Time For All Transactions [ms] [for Run] Minimum Elapsed Time For All Transactions [ms] [for Run] Total Transactions Completed [for Run] 450 Total Transactions Started [for Run] 450 Elapsed Time Standard Deviation For All Transactions [for Run]

72,376 189,890 12,609

48,274.9

Screen Summary Total VPs Passed [for Run] 147 Total VPs Failed [for Run] 0 Total VPs Error [for Run] 3 Total SAP Screens Started [for Run] 750 Total SAP Screens Completed [for Run] 619 SAP Screens Completed Percent [for Run] 82.5 Average Screen Request Response Time For All SAP Screens [ms] [for Run] 718.6 Maximum Screen Request Response Time For All SAP Screens [ms] [for Run] 12,671 Minimum Screen Request Response Time For All SAP Screens [ms] [for Run] 15 Screen Request Response Time Standard Deviation For All SAP Screens [ms] [for Run] 1,460.7 SAP Screens Started Rate [per second] [for Run] 0.7 Percent VPs Passed [for Run] 98 10.2.3 Run Summary Service: Executed Test SC_1_2_3_4_5_6 Active Users 0 Completed Users 1,000 Total Users 1,000 Elapsed Time [H:M:S] 0:17:52 Run Status Complete Displaying Results for Computer: All Hosts Request Summary Average Response Time For All Requests [ms] [for Run] 660.9 Maximum Response Time For All Requests [ms] [for Run] 3,485 Minimum Response Time For All Requests [ms] [for Run] 50 Response Time Standard Deviation For All Requests [ms] [for Run] 868.6 Maximum Rate Request Started [per second] [for Run] 20.8 Minimum Rate Request Started [per second] [for Run] 0 Maximum Rate Request Success [per second] [for Run] 18.8
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 75 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Minimum Rate Request Success [per second] [for Run] Maximum Rate Request Fail [per second] [for Run] Minimum Rate Request Fail [per second] [for Run] Maximum Rate Request Timeout [per second] [for Run] Minimum Rate Request Timeout [per second] [for Run]

0 0 0 0 0

Bytes Summary Average Sent Bytes For All Requests [Bytes] [for Run] 1,148 Rate Sent Bytes [per second] [for Run] 1,392.7 Maximum Sent Bytes [per Interval] [for Run] 215,824 Minimum Sent Bytes [per Interval] [for Run] 0 Total Sent Bytes [for Run] 1,492,400 Average Received Bytes For All Requests [Bytes] [for Run] 642 Rate Received Bytes [per second] [for Run] 778.9 Maximum Received Bytes [per Interval] [for Run] 120,696 Minimum Received Bytes [per Interval] [for Run] 0 Total Received Bytes [for Run] 834,600 10.3 A: Rational Performance Tester Result Screens on one x3850 X5 10.3.1 Run Summary Executed Test SC_1_2_3_4_5_6 Active Users 0 Completed Users 1,000 Total Users 1,000 Elapsed Time [H:M:S] 0:16:06 Run Status Complete Displaying Results for Computer: All Hosts Page Summary Total Page Attempts [for Run] 5,640 Total Page Hits [for Run] 5,640 Average Response Time For All Pages [ms] [for Run] 9,075.5 Response Time Standard Deviation For All Pages [for Run] 22,191.2 Maximum Response Time For All Pages [ms] [for Run] 179,153 Minimum Response Time For All Pages [ms] [for Run] 0 Transaction Summary Average Elapsed Time For All Transactions [ms] [for Run] Maximum Elapsed Time For All Transactions [ms] [for Run] Minimum Elapsed Time For All Transactions [ms] [for Run] Total Transactions Completed [for Run] 450 Total Transactions Started [for Run] 450 Elapsed Time Standard Deviation For All Transactions [for Run] 10.3.2 SAP Summary Executed Test SC_1_2_3_4_5_6 Active Users 0 Completed Users 1,000 Total Users 1,000
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 76 of 120

70,211.7 192,015 14,985

47,671.3

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Elapsed Time [H:M:S] 0:16:06 Run Status Complete Displaying Results for Computer: All Hosts Total SAP Connection Errors [for Run] 26 Transaction Summary Average Elapsed Time For All Transactions [ms] [for Run] Maximum Elapsed Time For All Transactions [ms] [for Run] Minimum Elapsed Time For All Transactions [ms] [for Run] Total Transactions Completed [for Run] 450 Total Transactions Started [for Run] 450 Elapsed Time Standard Deviation For All Transactions [for Run] 70,211.7 192,015 14,985

47,671.3

Screen Summary Total VPs Passed [for Run] 120 Total VPs Failed [for Run] 0 Total VPs Error [for Run] 26 Total SAP Screens Started [for Run] 750 Total SAP Screens Completed [for Run] 518 SAP Screens Completed Percent [for Run] 69.1 Average Screen Request Response Time For All SAP Screens [ms] [for Run] 342.2 Maximum Screen Request Response Time For All SAP Screens [ms] [for Run] 6,062 Minimum Screen Request Response Time For All SAP Screens [ms] [for Run] 0 Screen Request Response Time Standard Deviation For All SAP Screens [ms] [for Run] 697.1 SAP Screens Started Rate [per second] [for Run] 0.78 Percent VPs Passed [for Run] 82.2 10.3.3 Run Summary Service Executed Test SC_1_2_3_4_5_6 Active Users 0 Completed Users 1,000 Total Users 1,000 Elapsed Time [H:M:S] 0:16:06 Run Status Complete Displaying Results for Computer: All Hosts Request Summary Average Response Time For All Requests [ms] [for Run] 187 Maximum Response Time For All Requests [ms] [for Run] 975 Minimum Response Time For All Requests [ms] [for Run] 35 Response Time Standard Deviation For All Requests [ms] [for Run] Maximum Rate Request Started [per second] [for Run] 13.9 Minimum Rate Request Started [per second] [for Run] 0 Maximum Rate Request Success [per second] [for Run] 13.7 Minimum Rate Request Success [per second] [for Run] 0 Maximum Rate Request Fail [per second] [for Run] 0 Minimum Rate Request Fail [per second] [for Run] 0 Maximum Rate Request Timeout [per second] [for Run] 0
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

207.2

Page 77 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Minimum Rate Request Timeout [per second] [for Run]

Bytes Summary Average Sent Bytes For All Requests [Bytes] [for Run] 1,148 Rate Sent Bytes [per second] [for Run] 1,544.8 Maximum Sent Bytes [per Interval] [for Run] 157,276 Minimum Sent Bytes [per Interval] [for Run] 0 Total Sent Bytes [for Run] 1,492,400 Average Received Bytes For All Requests [Bytes] [for Run] 642 Rate Received Bytes [per second] [for Run] 863.9 Maximum Received Bytes [per Interval] [for Run] 87,954 Minimum Received Bytes [per Interval] [for Run] 0 Total Received Bytes [for Run] 834,600 10.4 A: VMware vSphere Virtual Center Overview about the Landscape in the left navigation bar

Figure 78

Overview about the physical system IBM1

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 78 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 79 Overview about the physical system IBM2

Figure 80 Virtual Machine configuration

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 79 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

VM configuration for SID BI7 (BW) PI7 (PI) EP7 (EP) R61 (ECC PRD) R62 (ECC QA) R63 (ECC Dev) SM1 Solution Manager anvm04_BI_BJ7 CRMA 700 SRMA 701 NWAB 702 NWAB-711

Virtual CPUs 4 8 8 8 8 8 4 4 4 4 4 4

CPU reserved (GHz) 1 1 1. 9.59 1 1 1 0.5 0 0 0 0

Memory reserved (GB) 1 1 1 0.629 1 1 1 1 0 0 0 0

Memory Limit (GB) 9 17,4 9 17.6 9 9 9 9 4 4 4 2

Memory configured (GB) 8 16 8 16 8 8 8 8 4 4 4 2

Table 2: defined VMs with the respective CPU and memory settings

Figure 81

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 80 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 82: Overview about the VMs in the landscape

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 81 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Guest OS: SUSE Linux VMs Release SUSE Linux Enterprise Server 10 (x86_64) VERSION = 10 PATCHLEVEL = 2 Filesystems
R61: Filesystem /dev/sda1 udev /dev/sdb1 R62: Filesystem /dev/sda1 udev /dev/sdb1 R63: Filesystem /dev/sda1 udev /dev/sdb1 PI7: Filesystem /dev/sda1 udev /dev/sdb1 BI7: Filesystem /dev/sda1 udev /dev/sdb1 SM1: Filesystem /dev/sda1 udev /dev/sdb1 EP7: Filesystem /dev/sda1 udev /dev/sdb1 Size 48G 72G 378G Size 48G 66G 378G Size 48G 66G 378G Size 48G 72G 95G Size 48G 66G 95G Size 16G 66G 95G Size 48G 66G 48G Used Avail Use% Mounted on 4.2G 41G 10% / 3.8G 69G 6% /dev 287G 73G 80% /coedata Used Avail Use% Mounted on 5.6G 40G 13% / 717M 66G 2% /dev 325G 34G 91% /coedata Used Avail Use% Mounted on 3.8G 42G 9% / 633M 66G 1% /dev 263G 97G 74% /coedata Used Avail Use% Mounted on 7.9G 37G 18% / 252M 72G 1% /dev 45G 46G 50% /coedata Used Avail Use% Mounted on 3.9G 41G 9% / 17G 50G 26% /dev 55G 36G 61% /coedata Used Avail Use% Mounted on 4.0G 11G 27% / 150M 66G 1% /dev 62G 28G 70% /coedata Used Avail Use% Mounted on 3.7G 42G 9% / 100K 66G 1% /dev 20G 26G 43% /coedata

D70, C70, N70 and N71 are running on Windows Server 2003 x86_64 SP2

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 82 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.5 A: XIV: Layout and Naming Convention The following table shows the storage layout:
Name DATA_BJ_BJ7 DATA_ECC_R62 DATA_ECC_R63 DB_BI_BI7 DB_ECC_R61 DB_EP_EP7 DB_PI_PI7 DB_SM_SM1 LOG_BI_BI7 LOG_ECC_R61 LOG_EP_EP7 LOG_PI_PI7 LOG_SM_SM1 OS_BI_BI7 OS_BJ_BJ7 OS_ECC_R61 OS_ECC_R62 OS_ECC_R63 OS_EP_EP7 OS_PI_PI7 OS_SM_SM1 SWAP_BI_BI7 SWAP_BJ_BJ7 SWAP_ECC_R61 SWAP_ECC_R62 SWAP_ECC_R63 SWAP_EP_EP7 SWAP_PI_PI7 SWAP_SM_SM1 VMWARE_VMFS vmfs_shared Size (GB) 68 412 412 103 412 51 103 103 17 51 17 17 17 51 51 51 51 51 51 51 17 85 85 85 85 85 85 85 85 515 704 Pool X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86 X86

Table 3: Layout on XIV

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 83 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.6 A: Network infrastructure product ECC 6.04 details hostname central instance DB hostname central instance DB hostname central instance DB hostname central instance SCS DB hostname central instance DB hostname central instance SCS DB hostname JC SCS DB hostname hostname hostname hostname SID R61 hostname anvm62 cir61 dbr61 anvm101 cir62 dbr62 anvm102 cir63 dbr63 anvm40 cipi7 scspi7 dbpi7 anvm06 cibi7 dbbi7 anvm41 cism1 scssm1 dbsm1 anvm57 jcep7 scsep7 dbep7 anvm38 anvm55 anvm67 anvm31 Stack ABAP

ECC 6.04

R62

ABAP

ECC 6.04

R63

ABAP

PI 7.11

PI7

ABAP + Java

BI 7.01 ABAP

BI7

ABAP

SolutionManager 7.0

SM1

ABAP + Java

Portal 7.01

EP7

Java

SRM 7.0 CRM 7.0 NW 7.02 NW 7.11

D70 C70 N70 N71

ABAP ABAP ABAP ABAP

Table 4: SAP Components all as 2-tier, central systems, hostnames and SAP kernel stack setup The standard procedure of installing SAP systems in the SAP Center of Excellence is to use saphostagents according to the implementation guidelines of SAP Adaptive Computing. For details, please check: http://www.sdn.sap.com/irj/sdn/adaptive .

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 84 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.7 A: Backup / Recovery / Restore This section describes the different backup methods provided by Tivoli Storage Manager (TSM) 10.7.1 A: Data protection with Tivoli Storage Manager for ERP In order to backup SAP the DB2 user interface (CLI) is used, TSM for ERP integrates with DB2. The DB2 BACKUP DATABASE command performs this DB2 server process: Creates a unique timestamp to identify the backup Loads Data Protection for SAP dynamically as a shared library Reads the data from the database containers Reads the DB2 configuration files Creates data blocks containing the backup image and passes these blocks to the data mover part of Data Protection for SAP TSM for ERP sends the data to the Tivoli Storage Manager storage system whether tape or disk. At the end of the backup process, the DB2 server process logs the backup in the recovery history file. The DB2 RESTORE DATABASE command performs this DB2 server process: Loads Data Protection for SAP dynamically as a shared library Requests the backup data from the shared library Retrieves the data blocks from TSM, if available Passes the data blocks to the DB2 Server Process The DB2 server process restores the DB2 data to the database containers The DB2 server process logs the restore in the recovery history file
10.7.1.1 Installing TSM for ERP (Data Protection for SAP) for DB2 on Linux

Data Protection for SAP for DB2 on Linux comes as a single executable file for each platform. Packages on the FTP server contain 'FTP' prior to the platform designation. For a disc or disc image, the name has the format: version-TIV-TSMERPDB2-platform When the file is launched, Data Protection for SAP guides you through the installation procedure. Read the descriptions carefully and follow the guidelines on the panels. Perform the following tasks to install Data Protection for SAP on the Linux system: 1. Log in as the root user on the SAP database server machine. 2. Verify that the DISPLAY variable is set to view the installation prompts through a graphical X-Window. 3. Start the DB2 instance. The installation program makes the necessary updates to the DB2 configuration. 4. Invoke the Data Protection for SAP executable file and follow the installation prompts. 5. View the summary in the last page of the installation wizard. The summary displays the Data Protection for SAP installation path where the installation log file (log.txt) is located. IBM SAP International Competence Center, Walldorf, Germany Page 85 of 120
2010 Global Alliance Solutions, SAP

IBM SAP Implementation and Operation in Integrated Environments Version 1.0 10.7.1.2 Configuring TSM for ERP for DB2

Data Protection for SAP for DB2 requires certain configuration tasks, which need to be performed for these applications: Data Protection for SAP base product Administration Assistant DB2 Log Manager and related DB2 files Tivoli Storage Manager backup-archive client Tivoli Storage Manager server Optional: HACMP For details please refer to the installation guide.
10.7.1.3 Implementing the Strategy by Scheduling Automated Backup Runs

Scheduling (or automating) backup and archive operations help ensure that the data is backed up regularly at a specified time. These products provide scheduled operations: SAP scheduler The SAP Computer Center Management System (CCMS) provides a scheduler for database administration and backup planning on a single database server. The scheduler can be started from the SAP GUI command line (transaction code db13) or with the SAP GUI menu function Tools ->CCMS -> DB administration -> DBA scheduling. Scheduler Crontab Automating backups at the database server level is available using either the the crontab command (for UNIX or Linux) or the schedule services feature (on Windows). Tivoli Storage Manager scheduler Tivoli Storage Manager also provides a scheduler function for all of its clients. As a result, automation can be performed for multiple database servers. The Tivoli Storage Manager administrative client GUI provides a user-friendly wizard for defining schedules. Information on how to define Tivoli Storage Manager schedules can be found in the Tivoli Storage Manager Administrator's Reference manual. IBM Tivoli Workload Scheduler (not used in this PoC) The IBM Tivoli Workload Scheduler provides event-driven automation, monitoring, and job control for both local and remote systems. More information can be found at http://www.ibm.com/software/tivoli/products/scheduler/. Sample Backup Strategy for Daily Backup Processing This Figure 83 illustrates the sequence of backup operations to consider for a daily backup schedule.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 86 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 83: Sequence of backup operations The automated backup example (shown in the figure above) displays these common tasks: A full database backup (offline or without application load) performed each night. Offline log files are backed up to disk during online hours. This has the advantage of eliminating the need for extra tape mounts for relatively small files. The Tivoli Storage Manager server migrates archived log files from disk to tape after the full database backup. SAP system files are backed up incrementally with the Tivoli Storage Manager backup-archive client. The last backup in the daily cycle is the backup of the Tivoli Storage Manager database. This should always be performed. Backups can be performed to disk storage as well as to tape media. The Tivoli Storage Manager server manages the data regardless of the storage media. However, backing up the SAP database directly to tape is the preferred choice.
10.7.1.4 Starting Backups in a Non-Partitioned Database Environment

The following examples show how you can start DB2 database/tablespace backups from the command line using DB2 CLP. To start a DB2 backup or restore with Data Protection for SAP for DB2, log on as user db2SID or SIDadm. In these examples, the variable shared library represents the full path of the Data Protection for SAP shared library. DB2 database and tablespace backups are performed as follows: Full online backup (database parameter LOGRETAIN has to be activated): db2 backup db dbname online load shared library Online tablespace backup (database parameter LOGRETAIN has to be activated): db2 backup db dbname online tablespace (tablespace_name#1, ...) load shared library
10.7.1.5 Starting Restores in a Non-Partitioned Database Environment

The following examples show how you can start DB2 database/tablespace restores from the command line using DB2 CLP. Every successful backup run generates a timestamp that is required for later restore operations. These timestamps are written to the DB2 recovery IBM SAP International Competence Center, Walldorf, Germany Page 87 of 120
2010 Global Alliance Solutions, SAP

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

history file (RHF), which can be queried with DB2 commands. The timestamps of backup images currently stored on the Tivoli Storage Manager server can be queried using the Backup Object Manager query commands. If no timestamp is specified in a restore command, the latest backup image found on Tivoli Storage Manager will be restored. DB2 database and tablespace restores are performed as follows: Full restore to a certain point in time: db2 restore db dbname load shared library taken at timestamp Online tablespace restore db2 restore db dbname tablespace (tablespace_name#1, ...) online load shared library taken at timestamp Recovery History File restore db2 restore db dbname history file online load shared library Data Protection for SAP for DB2 process results can be checked by analyzing the Data Protection for SAP log files. These log files may contain success, warning, and error messages. Please refer to the documentations for TSM concepts and implementations mentioned in the References chapter (9.3). 10.7.2 A: Data protection with IBM FlashCopy Manager (FCM) The IBM FlashCopy Manager provides the functionality to perform a backup based on FlashCopy. With Version 2.2 FCM supports the LINUX x86_64 operation system as well as the IBM storage system XIV. How to use FCM for this environment has been documented in this whitepaper, including backup, restore, and cloning. http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101703 This section provides additional information how to use FCM in combination with VMware, the discovery of flash copied LUNs and how to access them from LINUX as VM is different comparing if LINUX runs on the server natively. The only process derivation from the standard process during the FlashCopy based backup of SAP within VMware is the mount operation of the FlashCopy volumes by the backup (proxy) node. Major steps to perform a FlashCopy backup: 1. 2. 3. 4. 5. 6. 7. Prepare LINUX systems Create FlashCopy Backup (DB2 tool) Identify the new FlashCopy LUNs on XIV (LINUX tools) Map the FlashCopy LUNs to ESX Server (XIV CLI) Provide LUN access from ESX server to LINUX (VMware GUI or CLI) Scan SCSI bus on LINUX (LINUX tools) Run FCM in order to process the FlashCopy backup

Steps 3 to 6 are in addition to the standard procedure in order to adapt to a VMware environment.
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 88 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

All commands are executed on the production server, if not stated otherwise.
10.7.2.1 Prepare LINUX systems

Install the XIV Host Attachment kit on both LINUY systems, the production (Prod) and the backup (BA) system, use V 1.5.2 or higher Install the XIV CLI on the Prod system, use V 2.4.3 or higher. Enable the LINUX standard multipath (MP) driver on BA server, run as root user: # /etc/init.d/boot.multipath start # /etc/init.d/multipathd start # insserv boot.multipath multipathd Install FCM according the product documentation and Whitepaper

10.7.2.2 Create FlashCopy backup

Run as DB2 user:

# db2 backup db <sid> online use snapshot

Verify backup # db2acsutil query status

Instance ========== db2r61 db2r61 db2r61 db2r61 db2r61 Database Part Image Time Status ======== ==== ============== ======================================== R61 0 20100811150653 Repetitively restorable + Tape backup pending (0 / 0 MB) R61 0 20100809112557 Repetitively restorable R61 0 20100805132010 Repetitively restorable (0 / 0 MB) R61 0 20100713161218 Repetitively restorable (0 / 0 MB) R61 0 20100712165421 Repetitively restorable (0 / 0 MB)

10.7.2.3 Identify the new FlashCopy LUNs on XIV

FCM stores the backup information in its own repository:

<path>/db2/db2<sid>/acs/acsrepository/

Identify the newest repository:

dbr61:db2r61 -rw-r--r-- 1 -rw-r--r-- 1 -rw-r--r-- 1 -rw------- 1 131> ls -l db2r61 dbr61adm 15554 2010-08-12 13:33 ori.1281600031000 db2r61 dbr61adm 15554 2010-08-12 14:14 ori.1281612783468 db2r61 dbr61adm 15566 2010-08-12 15:56 ori.1281615193969 db2r61 dbr61adm 132 2010-08-12 09:53 root

Identify the master prefix of FCM:

# cat root This directory contains a TSM repository. Do not edit. int=1 string=26,snapshot backup repository int=48 int=2 string=10,A0GCRBBRSG

Run this command sequence:

# grep -a "TSM_A0GCRBBRSG.*_1" ori.1281615193969 | awk TSM_A0GCRBBRSG_A0GCRKF9TX_01000171_1 TSM_A0GCRBBRSG_A0GCRKF9TX_01000172_1 TSM_A0GCRBBRSG_A0GCRKF9TX_01000378_1 -F \, '{print $2}'

The FlashCopy backup created 3 LUNs in our environment.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 89 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0 10.7.2.4 Map these FlashCopy LUNs to the ESX Server by using the XIV CLI xcli -u admin -p <password> -m <XIV IP address> -y map_vol vol=<Volume name> host=<ESX server> lun=<LUN number> # /opt/XIVGUI/xcli -u admin -p xxx -m 172.17.226.167 -y map_vol vol=TSM_A0GCRBBRSG_A0GCRKF9TX_01000171_1 host=IBMBlade11 lun=4 # /opt/XIVGUI/xcli -u admin -p xxx -m 172.17.226.167 -y map_vol vol=TSM_A0GCRBBRSG_A0GCRKF9TX_01000172_1 host=IBMBlade11 lun=5 # /opt/XIVGUI/xcli -u admin -p xxx -m 172.17.226.167 -y map_vol vol=TSM_A0GCRBBRSG_A0GCRKF9TX_01000378_1 host=IBMBlade11 lun=6

Hint: The parameter <host> does not allow to have a _ (underline) or (hyphen) in the name.
10.7.2.5 Provide LUN access from ESX server to LINUX

Discover LUNs at the ESX Server with the VMware vSphere GUI: Select the ESX server Select tap Configuration run Rescan

Figure 84

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 90 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 85 Provide LUN access to the LINUX system: Select the LINUX server Edit Settings Select tap Hardware Raw Device Mapping

Add

Hard Disk

Figure 86

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 91 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 87

Figure 88

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 92 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 89

Figure 90

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 93 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 91

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 94 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.7.2.6 Scan SCSI bus on LINUX (LINUX tools)

On the backup system:

# /bin/rescan-scsi-bus.sh Host adapter 0 (mptspi) found. Scanning SCSI subsystem for new devices Scanning host 0 channels 0 for SCSI target IDs 0 1 2 3 4 5 6 7, all LUNs Scanning for device 0 0 0 0 ... OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: VMware Model: Virtual disk Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 02 Scanning for device 0 0 1 0 ... NEW: Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: IBM Model: 2810XIV Rev: 10.2 Type: Direct-Access ANSI SCSI revision: 05 Scanning for device 0 0 2 0 ... NEW: Host: scsi0 Channel: 00 Id: 02 Lun: 00 Vendor: IBM Model: 2810XIV Rev: 10.2 Type: Direct-Access ANSI SCSI revision: 05 Scanning for device 0 0 3 0 ... NEW: Host: scsi0 Channel: 00 Id: 03 Lun: 00 Vendor: IBM Model: 2810XIV Rev: 10.2 Type: Direct-Access ANSI SCSI revision: 05 0 new device(s) found. 0 device(s) removed. anvm62:/ # xiv_devlist XIV Devices ---------------------------------------------------------------------Device Size Paths Vol Name Vol Id XIV Id XIV Host ---------------------------------------------------------------------/dev/sda 51.5GB N/A OS_ECC_R61 188 1300256 BigX-Server1 ---------------------------------------------------------------------/dev/sdb 412.3GB N/A DATA_ECC_R61 191 1300256 BigX-Server1 ---------------------------------------------------------------------/dev/sdc 85.9GB N/A SWAP_ECC_R61 190 1300256 BigX-Server1 ---------------------------------------------------------------------/dev/sdd 1.0TB N/A DB_ECC_R61 369 1300256 BigX-Server1 ---------------------------------------------------------------------/dev/sde 51.5GB N/A LOG_ECC_R61 370 1300256 BigX-Server1 ---------------------------------------------------------------------/dev/sdf 601.3GB N/A DB_ECC_R61_2 888 1300256 BigX-Server1 ----------------------------------------------------------------------

10.7.2.7 Run FCM in order to process the FlashCopy backup

tsm4acs -f mount dbr61:db2r61 135> /coedata/db2/db2r61/sqllib/acs/tsm4acs -f mount IBM Tivoli Storage FlashCopy(R) Manager offload agent - Version 2, Release 2, Level 0.0 for Linux x86_64 Build: 474 generated on May 19 2010 (c) Copyright IBM Corporation, 2000, 2010, All Rights Reserved. FMM0005I FMM8800I FMM6501I FMM6502I FMM6555I FMM6503I FMM6512I FMM0020I FMM0021I FMM0024I Start of program at: Thu 26 Aug 2010 11:17:22 AM CEST. The command is: mount Initializing 'mount' request. Executing 'mount' request. Selected snapshot backup with ID 'db2r61,R61,20100812140824'. Terminating 'mount' request. The 'mount' request for database 'R61' with partitions (0) processed successfully. End of program at: Thu 26 Aug 2010 11:18:07 AM CEST. Elapsed time: 45 sec. Return code is: 0.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 95 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

To verify that the FlashCopy volumes are mounted on the backup server run the mount command at the backup server:
# mount /dev/sda2 on / type ext3 (rw,acl,user_xattr) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) udev on /dev type tmpfs (rw,size=189g) devpts on /dev/pts type devpts (rw,mode=0620,gid=5) securityfs on /sys/kernel/security type securityfs (rw) /dev/mapper/sapdatavg_A0GCRKLANE-sapinstance on /db2/R61 type ext3 (rw,data=ordered) /dev/mapper/saplogvg_A0GCRKLAEW-saploglv on /db2/R61/log_dir type ext3 (rw,data=ordered) /dev/mapper/sapdatavg_A0GCRKLANE-sapdata1lv on /db2/R61/sapdata1 type ext3 (rw,data=ordered) /dev/mapper/sapdatavg_A0GCRKLANE-sapdata2lv on /db2/R61/sapdata2 type ext3 (rw,data=ordered) /dev/mapper/sapdatavg_A0GCRKLANE-sapdata3lv on /db2/R61/sapdata3 type ext3 (rw,data=ordered) /dev/mapper/sapdatavg_A0GCRKLANE-sapdata4lv on /db2/R61/sapdata4 type ext3 (rw,data=ordered)

In order to copy the data to the TSM server, or to restore the production data through FCM restore functionality refer to the FCM product documentation; both tasks are independent to VMware.
10.7.2.8 Troubleshooting

After a successful tsm4acs -mount operation, and copying the data to the TSM server, it is necessary to run tsm4acs -f unmount If tsm4acs -mount fails typically RC of 2 is displayed, it is required to run tsm4acs before the next mount operation, sequence:
tsm4acs mount tsm4acs unmount /bin/rescan-scsi-bus.sh (on BA node) -f unmount

In case of an error read the summary.<time stamp> log file. If this kind of error msg has been logged:
MNT 15:43:11 (43806940) FMM0854E During the mount operation IBM Tivoli Storage FlashCopy(R) Manager identified that the following target volumes are not visible on the backup or cloning system 'anvm142'. List of missing target volumes: 010003D2 010003D3 010003D4

Then you need to execute these commands:

tsm4acs unmount /bin/rescan-scsi-bus.sh (on BA node) tsm4acs mount

In case this kind of error msg has been logged:

MNT 15:25:23 (43806940) FMM4003E Operation mount will be aborted. Physical volume(s) with conflicting physical volume id(s) have been found on the target host 'anvm142' C2TAUe-u6yl-0G3G-FzuL-Kl1S-y02G-Xfk9SE (/dev/sdd) 2MT4Ga-LoO2-pCWe-D9Uv-hc4L-NLR3-7l63oO (/dev/sdc) YqhPbQ-AEk6-glPQ-vgFs-meGN-M8PB-5icYuG (/dev/sdb)

Then you need to execute these commands:

tsm4acs unmount On BA node, for all /dev/sd<X> devices as listed in log file: /bin/rescan-scsi-bus.sh

echo 1 > /sys/block/sdd/device/delete echo 1 > /sys/block/sdc/device/delete echo 1 > /sys/block/sdb/device/delete

tsm4acs mount

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 96 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.8 A: Details on Cloning Procedure The following screenshots describe best how to clone an SAP system using the IBM SAP cloning solution, step by step:

Figure 92: Create a new VM with a new name

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 97 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 93: Define the shared storage location

Figure 94: The new VM usues VMware 7 image format

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 98 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 95: Select guest operating system

Figure 96: Select the number of virtual processors

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 99 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 97: Assign memory parameters

Figure 98: Assign the isolated network

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 100 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 99: Selection of the SCSI controller

Figure 100: Select the LUNS which are actually snapshots taken on the XIV
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 101 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 101: Select a datastore

Figure 102: Selection of the compatibility mode, here physical is used

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 102 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 103: Selection of the first internal disk (SCSI port)

Figure 104: Summary screen, how the new VM is defined

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 103 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 105: View of edit VM: the disks are assigned Attention: Please adjust the files /boot/grub/grub.conf and /etc/fstab accordingly to ensure disks are assigned properly.

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 104 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 106: On the XIV GUI you see the list of available snapshots, configured as read/write

Figure 107: XIV GUI assignment volume (LUN) to ESX host (Host Mapping)

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 105 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 108: vCenter: Available LUNs are visible in the configuration tab

Figure 109: vCenter: VMware is started and ready to use, with the same SID but isolated via VLAN settings

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 106 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Figure 110: The SAP system is up and running (saposcol)

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 107 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.9 A: DB2 Configuration All DB2 databases: DB21085I Instance "db2<sid>" uses "64" bits and DB2 code release "SQL09071" with level identifier "08020107". Informational tokens are "DB2 v9.7.0.1", "special_23487", "IP23034_23487", and Fix Pack "1". Product is installed at "/coedata/db2/db2<sid>/db2_software". R61: Database Configuration for Database R61 cir61:r61adm 54> db2 get db cfg for r61
Database configuration release level Database release level Database territory Database code page Database code set Database country/region code Database collating sequence IDENTITY_16BIT Alternate collating sequence Number compatibility Varchar2 compatibility Date compatibility Database page size Dynamic SQL Query management Statement concentrator Discovery support for this database = 0x0d00 = 0x0d00 = = = = = (ALT_COLLATE) = = = = = en_US 1208 UTF-8 1

OFF OFF OFF 16384

(DYN_QUERY_MGMT) = DISABLE (STMT_CONC) = OFF (DISCOVER_DB) = ENABLE = = = = = = = = NO 5 1 NO 0 SYSTEM 10 20

Restrict access Default query optimization class (DFT_QUERYOPT) Degree of parallelism (DFT_DEGREE) Continue upon arithmetic exceptions (DFT_SQLMATHWARN) Default refresh age (DFT_REFRESH_AGE) Default maintained table types for opt (DFT_MTTB_TYPES) Number of frequent values retained (NUM_FREQVALUES) Number of quantiles retained (NUM_QUANTILES) Decimal floating point rounding mode ROUND_HALF_EVEN Backup pending All committed transactions have been written to disk Rollforward pending Restore pending Multi-page file allocation enabled Log retain for recovery status User exit for logging status
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

(DECFLT_ROUNDING) = = NO = NO = NO = NO = YES = NO = NO
Page 108 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Self tuning memory (SELF_TUNING_MEM) Size of database shared memory (4KB) (DATABASE_MEMORY) AUTOMATIC(477160) Database memory threshold (DB_MEM_THRESH) Max storage for lock list (4KB) (LOCKLIST) AUTOMATIC(11712) Percent. of lock lists per application (MAXLOCKS) AUTOMATIC(97) Package cache size (4KB) (PCKCACHESZ) AUTOMATIC(128000) Sort heap thres for shared sorts (4KB) (SHEAPTHRES_SHR) AUTOMATIC(6105) Sort list heap (4KB) (SORTHEAP) AUTOMATIC(1221) Database heap (4KB) AUTOMATIC(2380) Catalog cache size (4KB) Log buffer size (4KB) Utilities heap size (4KB) Buffer pool size (pages) SQL statement heap (4KB) AUTOMATIC(4096) Default application heap (4KB) AUTOMATIC(256) Application Memory Size (4KB) AUTOMATIC(40000) Statistics heap size (4KB) AUTOMATIC(4384) Interval for checking deadlock (ms) Lock timeout (sec) Changed pages threshold Number of asynchronous page cleaners AUTOMATIC(7) Number of I/O servers AUTOMATIC(4) Index sort flag Sequential detect flag Default prefetch size (pages) Track modified pages Default number of containers Default tablespace extentsize (pages) Max number of active applications AUTOMATIC(74) Average number of active applications AUTOMATIC(3) Max DB files open per application Log file size (4KB) Number of primary log files Number of secondary log files Changed path to log files Path to log files /coedata/db2/R61/log_dir/NODE0000/ Overflow log path Mirror log path First active log file Block log on disk full

= ON = = 10 = = = = =

(DBHEAP) = (CATALOGCACHE_SZ) (LOGBUFSZ) (UTIL_HEAP_SZ) (BUFFPAGE) (STMTHEAP) = = = = = 2560 1024 10000 10000

(APPLHEAPSZ) = (APPL_MEMORY) = (STAT_HEAP_SZ) = (DLCHKTIME) = 10000 (LOCKTIMEOUT) = 3600 (CHNGPGS_THRESH) = 40 (NUM_IOCLEANERS) = (NUM_IOSERVERS) = (INDEXSORT) = YES (SEQDETECT) = YES (DFT_PREFETCH_SZ) = AUTOMATIC (TRACKMOD) = ON = 1 (DFT_EXTENT_SZ) = 2 (MAXAPPLS) = (AVG_APPLS) = (MAXFILOP) = 61440 (LOGFILSIZ) (LOGPRIMARY) (LOGSECOND) (NEWLOGPATH) = 16380 = 20 = 40 = =

(OVERFLOWLOGPATH) = (MIRRORLOGPATH) = = (BLK_LOG_DSK_FUL) = YES

Page 109 of 120

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Block non logged operations (BLOCKNONLOGGED) = NO Percent max primary log space by transaction (MAX_LOG) = 0 Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0 Group commit count (MINCOMMIT) Percent log file reclaimed before soft chckpt (SOFTMAX) Log retain for recovery enabled (LOGRETAIN) User exit for logging enabled (USEREXIT) HADR HADR HADR HADR HADR HADR HADR HADR HADR database role local host name local service name remote host name remote service name instance name of remote server timeout value log write synchronization mode peer window duration (seconds) (HADR_LOCAL_HOST) (HADR_LOCAL_SVC) (HADR_REMOTE_HOST) (HADR_REMOTE_SVC) (HADR_REMOTE_INST) (HADR_TIMEOUT) (HADR_SYNCMODE) (HADR_PEER_WINDOW) = = = = = = = = = = = = = 1 300 OFF OFF STANDARD

120 NEARSYNC 0 OFF OFF 5 20

First log archive method (LOGARCHMETH1) = Options for logarchmeth1 (LOGARCHOPT1) = Second log archive method (LOGARCHMETH2) = Options for logarchmeth2 (LOGARCHOPT2) = Failover log archive path (FAILARCHPATH) = Number of log archive retries on error (NUMARCHRETRY) = Log archive retry Delay (secs) (ARCHRETRYDELAY) = Vendor options (VENDOROPT) = Auto restart enabled (AUTORESTART) Index re-creation time and redo index build (INDEXREC) (RESTART) Log pages during index build (LOGINDEXBUILD) Default number of loadrec sessions (DFT_LOADREC_SES) Number of database backups to retain (NUM_DB_BACKUPS) Recovery history retention (days) (REC_HIS_RETENTN) Auto deletion of recovery objects (AUTO_DEL_REC_OBJ) TSM TSM TSM TSM management class node name owner password maintenance database backup table maintenance runstats statement statistics statistics profiling profile updates reorganization (TSM_MGMTCLASS) (TSM_NODENAME) (TSM_OWNER) (TSM_PASSWORD)

= ON = SYSTEM = = = = = = = = = OFF 1 12 60 OFF

Automatic Automatic Automatic Automatic Automatic Automatic Automatic Automatic

(AUTO_MAINT) = ON (AUTO_DB_BACKUP) = OFF (AUTO_TBL_MAINT) = ON (AUTO_RUNSTATS) = ON (AUTO_STMT_STATS) = ON (AUTO_STATS_PROF) = OFF (AUTO_PROF_UPD) = OFF (AUTO_REORG) = OFF (AUTO_REVAL) (CUR_COMMIT) (DEC_TO_CHAR_FMT) (ENABLE_XMLCHAR) (WLM_COLLECT_INT) (MON_REQ_METRICS) (MON_ACT_METRICS) (MON_OBJ_METRICS) (MON_UOW_DATA) (MON_LOCKTIMEOUT) = = = = = = = = = = DEFERRED ON NEW YES 0 BASE BASE BASE NONE NONE

Auto-Revalidation Currently Committed CHAR output with DECIMAL input Enable XML Character operations WLM Collection Interval (minutes) Monitor Collect Settings Request metrics Activity metrics Object metrics Unit of work events Lock timeout events

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 110 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Deadlock events WITHOUT_HIST Lock wait events Lock wait event threshold Number of package list entries Lock event notification level SMTP Server SQL conditional compilation flags Section actuals setting cir61:r61adm 55> db2 get dbm cfg

(MON_DEADLOCK) = (MON_LOCKWAIT) (MON_LW_THRESH) (MON_PKGLIST_SZ) (MON_LCK_MSG_LVL) = = = = NONE 5000000 32 1

(SMTP_SERVER) = (SQL_CCFLAGS) = (SECTION_ACTUALS) = NONE

Database Manager Configuration

Node type = Enterprise Server Edition with local and remote clients Database manager configuration release level CPU speed (millisec/instruction) 07 Communications bandwidth (MB/sec) 1.000000e+02 = 0x0d00 (CPUSPEED) = 1.889377e(COMM_BANDWIDTH) =

Max number of concurrently active databases (NUMDB) = 8 Federated Database System Support (FEDERATED) = NO Transaction processor monitor name (TP_MON_NAME) = Default charge-back account Java Development Kit installation path /db2/db2r61/sqllib/java/jdk64 (DFT_ACCOUNT_STR) = (JDK_PATH) = = 3 = 3 = = 0 ON ON ON ON ON ON ON = OFF = DBR61ADM = DBR61CTL = DBR61MNT = = = = = = UNFENCED = = =

Diagnostic error capture level (DIAGLEVEL) Notify Level (NOTIFYLEVEL) Diagnostic data directory path (DIAGPATH) /db2/R61/db2dump Size of rotating db2diag & notify logs (MB) (DIAGSIZE) Default database monitor switches Buffer pool (DFT_MON_BUFPOOL) = Lock (DFT_MON_LOCK) = Sort (DFT_MON_SORT) = Statement (DFT_MON_STMT) = Table (DFT_MON_TABLE) = Timestamp (DFT_MON_TIMESTAMP) = Unit of work (DFT_MON_UOW) = Monitor health of instance and databases (HEALTH_MON) SYSADM group name SYSCTRL group name SYSMAINT group name SYSMON group name (SYSADM_GROUP) (SYSCTRL_GROUP) (SYSMAINT_GROUP) (SYSMON_GROUP)

Client Userid-Password Plugin (CLNT_PW_PLUGIN) Client Kerberos Plugin (CLNT_KRB_PLUGIN) Group Plugin (GROUP_PLUGIN) GSS Plugin for Local Authorization (LOCAL_GSSPLUGIN) Server Plugin Mode (SRV_PLUGIN_MODE) Server List of GSS Plugins (SRVCON_GSSPLUGIN_LIST) Server Userid-Password Plugin (SRVCON_PW_PLUGIN) Server Connection Authentication (SRVCON_AUTH) NOT_SPECIFIED
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 111 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Cluster manager

(CLUSTER_MGR) = = = = = = = NO YES CLIENT NO

Database manager authentication (AUTHENTICATION) SERVER_ENCRYPT Alternate authentication (ALTERNATE_AUTH_ENC) NOT_SPECIFIED Cataloging allowed without authority (CATALOG_NOAUTH) Trust all clients (TRUST_ALLCLNTS) Trusted client authentication (TRUST_CLNTAUTH) Bypass federated authentication (FED_NOAUTH) Default database path Database monitor heap size (4KB) AUTOMATIC(90) Java Virtual Machine heap size (4KB) Audit buffer size (4KB) Size of instance shared memory (4KB) Backup buffer default size (4KB) Restore buffer default size (4KB) Agent stack size Sort heap threshold (4KB) Directory cache support

(DFTDBPATH) = /db2/R61 (MON_HEAP_SZ) = (JAVA_HEAP_SZ) (AUDIT_BUF_SZ) (INSTANCE_MEMORY) (BACKBUFSZ) (RESTBUFSZ) = = = = = 2048 0 680960 1024 1024

(AGENT_STACK_SZ) = 1024 (SHEAPTHRES) = 0 (DIR_CACHE) = NO

Application support layer heap size (4KB) (ASLHEAPSZ) = 16 Max requester I/O block size (bytes) (RQRIOBLK) = 65000 Query heap size (4KB) (QUERY_HEAP_SZ) = 1000 Workload impact by throttled utilities(UTIL_IMPACT_LIM) = 10 Priority of agents Agent pool size AUTOMATIC(100) Initial number of agents in pool Max number of coordinating agents AUTOMATIC(200) Max number of client connections AUTOMATIC(MAX_COORDAGENTS) Keep fenced process Number of pooled fenced processes AUTOMATIC(MAX_COORDAGENTS) Initial number of fenced processes (AGENTPRI) = SYSTEM (NUM_POOLAGENTS) = (NUM_INITAGENTS) = 5 (MAX_COORDAGENTS) = (MAX_CONNECTIONS) = (KEEPFENCED) = NO (FENCED_POOL) = (NUM_INITFENCED) = 0 (INDEXREC) = RESTART

Index re-creation time and redo index build Transaction manager database name Transaction resync interval (sec) SPM SPM SPM SPM name log size resync agent limit log path

(TM_DATABASE) = 1ST_CONN (RESYNC_INTERVAL) = 180 (SPM_NAME) (SPM_LOG_FILE_SZ) (SPM_MAX_RESYNC) (SPM_LOG_PATH) = = 256 = 20 =

TCP/IP Service name Discovery mode Discover server instance SSL server keydb file SSL server stash file SSL server certificate label
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

(SVCENAME) = sapdb2R61 (DISCOVER) = SEARCH (DISCOVER_INST) = ENABLE (SSL_SVR_KEYDB) = (SSL_SVR_STASH) = (SSL_SVR_LABEL) =

Page 112 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

SSL SSL SSL SSL SSL

service name cipher specs versions client keydb file client stash file

(SSL_SVCENAME) (SSL_CIPHERSPECS) (SSL_VERSIONS) (SSL_CLNT_KEYDB) (SSL_CLNT_STASH)

= = = = =

Maximum query degree of parallelism Enable intra-partition parallelism Maximum Asynchronous TQs per query

(MAX_QUERYDEGREE) = ANY (INTRA_PARALLEL) = NO (FEDERATED_ASYNC) = 0 = = = 10 = 5 = 60

No. of int. communication buffers(4KB)(FCM_NUM_BUFFERS) AUTOMATIC(4096) No. of int. communication channels (FCM_NUM_CHANNELS) AUTOMATIC(2048) Node connection elapse time (sec) (CONN_ELAPSE) Max number of node connection retries (MAX_CONNRETRIES) Max time difference between nodes (min) (MAX_TIME_DIFF) db2start/db2stop timeout (min)

(START_STOP_TIME) = 10

DB2 Registry Settings

dbr61:db2r61 52> db2set -all DB2DBDFT=R61 DB2_BLOCKING_WITHHOLD_LOBLOCATOR=NO [DB2_WORKLOAD] DB2_AGENT_CACHING_FMP=OFF [DB2_WORKLOAD] DB2_TRUST_MDC_BLOCK_FULL_HINT=YES [DB2_WORKLOAD] DB2_CREATE_INDEX_COLLECT_STATS=YES [DB2_WORKLOAD] DB2_ATS_ENABLE=YES [DB2_WORKLOAD] DB2_RESTRICT_DDF=YES [DB2_WORKLOAD] DB2_DUMP_SECTION_ENV=YES [DB2_WORKLOAD] DB2_OPT_MAX_TEMP_SIZE=10240 [DB2_WORKLOAD] DB2_WORKLOAD=SAP DB2_TRUNCATE_REUSESTORAGE=IMPORT [DB2_WORKLOAD] DB2_MDC_ROLLOUT=DEFER [DB2_WORKLOAD] DB2_ATM_CMD_LINE_ARGS=-include-manual-tables [DB2_WORKLOAD] DB2_SKIPINSERTED=YES [DB2_WORKLOAD] DB2_VIEW_REOPT_VALUES=YES [DB2_WORKLOAD] DB2_OBJECT_TABLE_ENTRIES=65532 [DB2_WORKLOAD] DB2_OPTPROFILE=YES [DB2_WORKLOAD] DB2_IMPLICIT_UNICODE=YES [DB2_WORKLOAD] DB2STMM=APPLY_HEURISTICS:YES [DB2_WORKLOAD] DB2_INLIST_TO_NLJN=YES [DB2_WORKLOAD] DB2_MINIMIZE_LISTPREFETCH=YES [DB2_WORKLOAD] DB2_REDUCED_OPTIMIZATION=4,INDEX,JOIN,NO_TQ_FACT,NO_HSJN_BUILD_FACT, STARJN_CARD_SKEW,NO_SORT_MGJOIN,CART OFF,CAP OFF [DB2_WORKLOAD] DB2NOTIFYVERBOSE=YES [DB2_WORKLOAD] DB2_INTERESTING_KEYS=YES [DB2_WORKLOAD] DB2_EVALUNCOMMITTED=YES [DB2_WORKLOAD] DB2_EXTENDED_OPTIMIZATION=NLJOIN_KEYCARD [DB2_WORKLOAD] DB2_ANTIJOIN=EXTEND [DB2_WORKLOAD] DB2COMPOPT=327685,704 [DB2_WORKLOAD] DB2ATLD_PORTS=60000:65000 DB2ENVLIST=INSTHOME SAPSYSTEMNAME dbs_db6_schema DIR_LIBRARY LD_LIBRARY_PATH DB2COUNTRY=1 DB2COMM=TCPIP [DB2_WORKLOAD] DB2SYSTEM=anvm62 DB2INSTDEF=db2r61

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 113 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Additional SAP Systems: D70, C70, N70 and N71 are running with MaxDB D70: MaxDB 7.7.04.28 C70: MaxDB 7.7.06.10 N70: MaxDB 7.8.01.09 N71: MaxDB 7.7.06.07

IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP

Page 114 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

10.10 A: SAP Instance and Default Profiles Exemplary ABAP Instance Profile from R61 #.******************************************************* #.* Instance profile R61_DVEBMGS00_CIR61 #.* * #.* Version = 000014 * #.* Generated by user = xxxx #.* Generated on = 27.05.2010 , 15:28:40 #.******************************************************** em/global_area_MB = 8000 em/address_space_MB = 10000 em/max_size_MB = 12000 SAPLOCALHOSTFULL = cir61.wdf.sap.corp icm/host_name_full = cir61.wdf.sap.corp login/no_automatic_user_sapstar = 0 korr_protected = 26629082080190936520 sapgui/user_scripting = TRUE rdisp/max_wprun_time = 3600 rdisp/keepalive_timeout = 300 icm/keep_alive_timeout = 300 rsdb/esm/buffersize_kb = 150000 rsdb/obj/max_objects = 40000 rsdb/obj/buffersize = 150000 abap/buffersize = 600000 zcsa/presentation_buffer_area = 24000000 rtbb/buffer_length = 200000 zcsa/db_max_buftab = 20000 sap/bufdir_entries = 10000 rsdb/cua/buffersize = 8500 PHYS_MEMSIZE = 8192 SAPSYSTEMNAME = R61 SAPSYSTEM = 00 INSTANCE_NAME = DVEBMGS00 DIR_CT_RUN = $(DIR_EXE_ROOT)/run DIR_EXECUTABLE = $(DIR_INSTANCE)/exe SAPLOCALHOST = cir61 exe/saposcol = $(DIR_CT_RUN)/saposcol rdisp/wp_no_dia = 48 rdisp/wp_no_btc = 6 enque/table_size = 20000 exe/icmbnd = $(DIR_CT_RUN)/icmbnd icm/server_port_0 = PROT=HTTP,PORT=80$$,TIMEOUT=600,PROCTIMEOUT=3600 #----------------------------------------------------------------------# SAP Message Server parameters are set in the DEFAULT.PFL #----------------------------------------------------------------------ms/server_port_0 = PROT=HTTP,PORT=81$$ rdisp/wp_no_enq = 1 rdisp/wp_no_vb = 3
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 115 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

rdisp/rfc_min_wait_dia_wp = 2 rdisp/wp_no_vb2 = 2 rdisp/wp_no_spo = 1 ssl/ssl_lib = $(DIR_EXECUTABLE)$(DIR_SEP)$(FT_DLL_PREFIX)sapcrypto$(FT_DLL) sec/libsapsecu = $(ssl/ssl_lib) ssf/ssfapi_lib = $(ssl/ssl_lib) login/accept_sso2_ticket = 1 login/create_sso2_ticket = 2 rsdb/ntab/entrycount = 60000 zcsa/table_buffer_area = 500000000 #--------------------------------------------------------------------#Values proposed by SAP for shared memory pool sizes #--------------------------------------------------------------------ipc/shm_psize_10 = 132000000 ipc/shm_psize_14 = 0 ipc/shm_psize_18 = 0 ipc/shm_psize_19 = 0 ipc/shm_psize_30 = 0 ipc/shm_psize_40 = 144000000 ipc/shm_psize_41 = 0 gw/max_conn = 2000 gw/max_sys = 1200 gw/max_overflow_size = 40000000 gw/max_shm_req = 200 rdisp/tm_max_no = 2000 rdisp/max_comm_entries = 1000

Exemplary Java Instance Profile from PI7 #.****************************************** #.* Instance profile PI7_DVEBMGS01_CIPI7 #.* * #.* Version = 000005 * #.* Generated by user = xxxx * #.* Generated on = 07.04.2010 , 13:18:29 * #.********************************************** SAPSYSTEMNAME = PI7 SAPSYSTEM = 01 INSTANCE_NAME = DVEBMGS01 DIR_CT_RUN = $(DIR_EXE_ROOT)/$(OS_UNICODE)/linuxx86_64 DIR_EXECUTABLE = $(DIR_INSTANCE)/exe SAPLOCALHOST = cipi7 SAPLOCALHOSTFULL = cipi7.wdf.sap.corp PHYS_MEMSIZE = 4096 DIR_PROFILE = $(DIR_INSTALL)/profile _PF = $(DIR_PROFILE)/PI7_DVEBMGS01_cipi7 SETENV_00 = DIR_LIBRARY=$(DIR_LIBRARY) SETENV_01 = LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH)
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 116 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

SETENV_02 = SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH) SETENV_03 = LIBPATH=$(DIR_LIBRARY):%(LIBPATH) SETENV_04 = PATH=$(DIR_EXECUTABLE):%(PATH) SETENV_05 = DB2_CLI_DRIVER_INSTALL_PATH=$(DIR_EXECUTABLE)/db6_clidriver #----------------------------------------------------------------------# Copy SAP Executables #----------------------------------------------------------------------Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) _CPARG0 = copy_links _CPARG1 = list:$(DIR_GLOBAL)/db6/LINUXX86_64/clidriver.lst _CPARG2 = source:$(DIR_GLOBAL)/db6/LINUXX86_64 Execute_01 = immediate $(DIR_CT_RUN)$(DIR_SEP)sapcpe$(FT_EXE) list:$(DIR_GLOBAL)$(DIR_SEP)bootstrap$(DIR_SEP)bootstrap.lst source:$(DIR_GLOBAL)$(DIR_SEP)bootstrap target:$(jstartup/DIR_CLUSTER)$(DIR_SEP)bootstrap pf=/usr/sap/PI7/SYS/profile/PI7_DVEBMGS01_cipi7 Execute_02 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0) $(_CPARG1) $(_CPARG2) _CPARG3 = list:$(DIR_GLOBAL)/db6/jdbc/jdbcdriver.lst _CPARG4 = source:$(DIR_GLOBAL)/db6/jdbc Execute_03 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG3) $(_CPARG4) _CPARG5 = list:$(DIR_CT_SAPJVM)/sapjvm_5.lst _CPARG6 = source:$(DIR_CT_SAPJVM) Execute_04 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG5) $(_CPARG6) _CPARG7 = list:$(DIR_GLOBAL)/bootstrap/bootstrap.lst _CPARG8 = source:$(DIR_GLOBAL)/bootstrap _CPARG9 = target:$(DIR_INSTANCE)/j2ee/cluster/bootstrap Execute_05 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG7) $(_CPARG8) $(_CPARG9) jstartup/trimming_properties = off SAPJVM_VERSION = 5.1.027 jstartup/vm/home = $(DIR_SAPJVM) jstartup/max_caches = 500 j2ee/dbdriver = $(DIR_EXECUTABLE)/db2jcc.jar:$(DIR_EXECUTABLE)/db2jcc_license_cu.jar #----------------------------------------------------------------------# Start ABAP database #----------------------------------------------------------------------_DB = db.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_06 = immediate rm -f $(_DB) Execute_07 = immediate ln -s -f $(DIR_CT_RUN)/startdb $(_DB) Start_Program_00 = immediate $(_DB) abap/buffersize = 600000 login/no_automatic_user_sapstar = 0 login/system_client = 001 login/create_sso2_ticket = 2 login/accept_sso2_ticket = 1
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 117 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

#ssl/ssl_lib = $(DIR_EXECUTABLE)/libsapcrypto.so #sec/libsapsecu = $(DIR_EXECUTABLE)/libsapcrypto.so #ssf/ssfapi_lib = $(DIR_EXECUTABLE)/libsapcrypto.so #ssf/name = SAPSECULIB exe/saposcol = $(DIR_CT_RUN)/saposcol rdisp/wp_no_dia = 13 48 # the number of dialog work processes was increased from 13 to 48 to achieve higher throughput rdisp/wp_no_btc = 6 rsdb/ntab/entrycount = 60000 rsdb/cua/buffersize = 8500 rsdb/obj/buffersize = 150000 rsdb/obj/max_objects = 40000 19.03.2010 10:13:51 rsdb/esm/buffersize_kb = 150000 exe/icmbnd = $(DIR_CT_RUN)/icmbnd rdisp/j2ee_start_control = 1 rdisp/j2ee_start = 1 rdisp/j2ee_libpath = $(DIR_EXECUTABLE) exe/j2ee = $(DIR_EXECUTABLE)/jstart$(FT_EXE) rdisp/j2ee_timeout = 600 rdisp/frfc_fallback = on rtbb/buffer_length = 200000 icm/HTTP/j2ee_0 = PREFIX=/,HOST=localhost,CONN=0-500,PORT=5$$00 icm/keep_alive_timeout = 300 icm/host_name_full = cipi7.wdf.sap.corp korr_protected = 26629082080190936520 #----------------------------------------------------------------------# Start SCSA administration #----------------------------------------------------------------------Execute_08 = local $(DIR_EXECUTABLE)/sapmscsa pf=$(_PF) -n #----------------------------------------------------------------------# Start SAP message server #----------------------------------------------------------------------_MS = ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_09 = local rm -f $(_MS) Execute_10 = local ln -s -f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS) Start_Program_01 = local $(_MS) pf=$(_PF) #----------------------------------------------------------------------# Start application server #----------------------------------------------------------------------_DW = dw.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_11 = local rm -f $(_DW) Execute_12 = local ln -s -f $(DIR_EXECUTABLE)/disp+work$(FT_EXE) $(_DW) Start_Program_02 = local $(_DW) pf=$(_PF) #----------------------------------------------------------------------# Start internet graphics server #----------------------------------------------------------------------IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 118 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

_IG = ig.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_13 = local rm -f $(_IG) Execute_14 = local ln -s -f $(DIR_EXECUTABLE)/igswd_mt $(_IG) Start_Program_03 = local $(_IG) -mode=profile pf=$(_PF) #----------------------------------------------------------------------# SAP Message Server parameters are set in the DEFAULT.PFL #----------------------------------------------------------------------ms/server_port_0 = PROT=HTTP,PORT=81$$ rdisp/wp_no_enq = 1 enque/table_size = 32000 rdisp/wp_no_vb = 3 rdisp/wp_no_vb2 = 2 rdisp/wp_no_spo = 1 rdisp/rfc_min_wait_dia_wp = 2 rdisp/keepalive_timeout = 300 rdisp/max_wprun_time = 3600 ipc/shm_psize_10 = 136000000 ipc/shm_psize_40 = 112000000 zcsa/table_buffer_area = 500000000 zcsa/db_max_buftab = 20000 zcsa/presentation_buffer_area = 24000000 sap/bufdir_entries = 10000 j2ee/instance_id = ID144213

Exemplary Java Start Profile from PI7 SAPSYSTEMNAME = PI7 SAPSYSTEM = 00 INSTANCE_NAME = SCS00 DIR_CT_RUN = $(DIR_EXE_ROOT)/$(OS_UNICODE)/linuxx86_64 DIR_EXECUTABLE = $(DIR_INSTANCE)/exe SAPLOCALHOST = scspi7 DIR_PROFILE = $(DIR_INSTALL)/profile _PF = $(DIR_PROFILE)/PI7_SCS00_scspi7 SETENV_00 = DIR_LIBRARY=$(DIR_LIBRARY) SETENV_01 = LD_LIBRARY_PATH=$(DIR_LIBRARY):%(LD_LIBRARY_PATH) SETENV_02 = SHLIB_PATH=$(DIR_LIBRARY):%(SHLIB_PATH) SETENV_03 = LIBPATH=$(DIR_LIBRARY):%(LIBPATH) SETENV_04 = PATH=$(DIR_EXECUTABLE):%(PATH) #----------------------------------------------------------------------# Copy SAP Executables #----------------------------------------------------------------------_CPARG0 = list:$(DIR_CT_RUN)/scs.lst Execute_00 = immediate $(DIR_CT_RUN)/sapcpe$(FT_EXE) pf=$(_PF) $(_CPARG0) OS_UNICODE = uc #----------------------------------------------------------------------# Start SAP message server #----------------------------------------------------------------------_MS = ms.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_01 = local rm -f $(_MS)
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 119 of 120

IBM SAP Implementation and Operation in Integrated Environments Version 1.0

Execute_02 = local ln -s -f $(DIR_EXECUTABLE)/msg_server$(FT_EXE) $(_MS) Restart_Program_00 = local $(_MS) pf=$(_PF) #----------------------------------------------------------------------# Start SAP enqueue server #----------------------------------------------------------------------_EN = en.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_03 = local rm -f $(_EN) Execute_04 = local ln -s -f $(DIR_EXECUTABLE)/enserver$(FT_EXE) $(_EN) Restart_Program_01 = local $(_EN) pf=$(_PF) #----------------------------------------------------------------------# Start gateway #----------------------------------------------------------------------_GW = gw.sap$(SAPSYSTEMNAME)_$(INSTANCE_NAME) Execute_05 = local rm -f $(_GW) Execute_06 = local ln -s -f $(DIR_EXECUTABLE)/gwrd$(FT_EXE) $(_GW) Restart_Program_02 = local $(_GW) pf=$(_PF) -no_abap #----------------------------------------------------------------------# SAP Message Server for Java #----------------------------------------------------------------------rdisp/msserv = 0 rdisp/msserv_internal = 3900 ms/standalone = 1 ms/server_port_0 = PROT=HTTP,PORT=81$$ #----------------------------------------------------------------------# SAP Enqueue Server #----------------------------------------------------------------------enque/serverinst = 00 rdisp/enqname = $(rdisp/myname) ipc/shm_psize_34 = 0 enque/async_req_max = 5000 enque/encni/threadcount = 4 enque/server/max_requests = 5000 enque/server/max_query_requests = 5000 enque/table_size = 64000 enque/snapshot_pck_ids = 1600

Special Parameters BW #.****************************************** setenv CPIC_MAX_CONV 1024 in the env. #para changes due to 316877 gw/max_conn = 2000 gw/max_sys = 1200 gw/max_overflow_size = 40000000 gw/max_shm_req = 200 rdisp/tm_max_no = 2000 rdisp/max_comm_entries = 1000
IBM SAP International Competence Center, Walldorf, Germany 2010 Global Alliance Solutions, SAP Page 120 of 120