Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative

Lessons Learned from Isbank –
A Story of a DB2 for z/OS initiative
Cüneyt Göksu
IBM DB2 Gold Consultant
VBT
Session Code: F13
Wednesday, 10 November 2010 | Platform: z/OS

1

Due to Isbank’s confidentiality regulations some of
the performance numbers, naming conventions and
figures are removed, rearranged or renamed.

2

Agenda

• How did we start?
• Installation, Parallel Sysplex and More
• Brand New Corebank Application and Java
• IMS & DB2 Coexistence
• Unicode, SQL and Application Tuning Experiences
• Moving Windows ODS Application to z/OS
• V9 Migration
• Thinking V10

3

How did we start?

• In 2007
• 2 x System z9 EC
• z/OS 1.7

• DB2 V8
• Omegamon for DB2 V4.1.0
• DB2 Admin Tool from IBM
• No Application!

4

Installations, Parallel Sysplex and More
• Sandbox LPAR, V8 Installation and Naming Conventions.
• Install DB2 V8 NFM with Codepage 1026 (Turkish)
• One-Way Data Sharing Enabled
• Two-Way Data Sharing...
• CF Sizing and DataSharing Failure Test Scenarios
• CF350 - DB2 for z/OS and Data Sharing Implementation
Workshop
• CF83 - DB2 for z/OS Database Admin Workshop
• CV721 - DB2 9 for z/OS Application Programming
Workshop
• CE130 – Advanced SQL Workshop
5

Test Environment
• 4 LPARs
• 5 DataSharing Groups
• 4 Members / Group WHY?!?!?!

DEV3 DEV4
DEV1 DEV2
TEST3 TEST4
TEST1 TEST2
PAT3 PAT4
PAT1 PAT2
UAT3 UAT4
UAT1 UAT2
PERF3 PERF4
PERF1 PERF2

LPAR1 LPAR1 LPAR1
LPAR1

6

Production Environment
• 4 LPARs
• 1 DataSharing Group
• 4 Members / Group

DEV1 Prod 2 Prod 3 Prod 4
Prod 1
TEST1
PAT1
UAT1
PERF1

LPAR1 LPAR1 LPAR1
LPAR1

7

Explore the DB2
• IpPlex / VIPA
• RTS and Utility Automation
• Dynamic SortWork Allocation for DB2 Utilities

PK45916: REMOVE THE NEED TO SPECIFY
SORTNUM ON UTILITY STATEMENTS FOR
DYNAMICALLY ALLOCATED SORT WORK DATA
SETS

• UTSORTAL = YES
IGNSORTN = YES
MAX_UTIL_PARTS  PK51853
8
• SORTWKnn, SW01WKnn, DATAWKnn, STATWKnn, ST01WKnn

Explore the DB2
Monitor and Action against critical DB2 messages such as
• DSNJ111E (OUT OF SPACE IN ACTIVE LOG DATA SETS)
• DSNI014I(Broken Page)
• DSNP007I(EXTEND FAILED)

SMFACCT=(1,2,3,7,8,10)
SMFSTAT=(1,3,4,5,6,8)
ACCUMACC = NO  Development-Performance
ACCUMACC = 10  Prod

SYSIBM.SYSPACKAGE WHERE OPERATIVE <> 'Y' OR
VALID <> 'Y'

PARMLIB(CSVLLAxx)  ....SDSNEXIT
/F LLA,UPDATE=xx
9

Explore the DB2
• “Automated Space Management” or “sliding secondary extent”
MGEXTSZ=YES, PriQTY = -1, SecQTY = -1

• CLI Packages (SYSxynzz)

x: L or S : package size
y: H or N : hold or nohold
n: 1,2,3,4 : 1=UR, 2=CS, 3=RS, 4=RR

SYSLH3xx, SYSLH4xx, SYSSH3xx, SYSSH4xx  DISABLED

odbc.ini (WithHold=1)  1: With HOLD, 0: No HOLD

10

Corebank Application & JAVA
• Insert ONLY Banking Application.
DELETEs and UPDATEs are logical.

• Member Cluster
• Trackmod NO
• FREEPAGE 0 & PCTFREE 10

• DB2 for z/OS
Optimizing INSERT Performance
John Campbell

Insert Performance in DB2 UDB for z/OS V7/V8
Akira Shibamiya
11

• Optimistic Locking
1. Read row
2. Modify data
3. Try to write back data
4. Before updating the row check whether it has been
changed in the meantime; if so return with error

• Lock Avoidance
Dirty Read on package level or statement level

12


• Date, Time, TimeStamp, Type Conversion Costs

• Timestamp  STRING Datatype in Java

• SELECT ..., CHAR (AAPCR_START_DATE),
CHAR (DIP_EFCTV_DATE), ..., CHAR (DIL_HOST_TS),
CHAR (DIL_LOCAL_TS), DIL_WS_ID, DIL_USER_ID,
RULE_SYSTEM_ID, LDBID
FROM DRVD_INTRST_LINE D
• 10,000 rows in table
13

• Avg CPU Cost of retrieving a row %1,3
with SQL Function is 0,000107026
without SQL Function is 0,000105629
107,500000000

107,025927708
107,000000000

106,500000000

CPU Usage with SQL Func
106,000000000
CPU Usage w/o SQL Func
105,628945059
105,500000000

105,000000000

104,500000000
1000000

14


15


• Beware!  PreTest Connection Properties
16


• SELECT ....FROM ...WHERE ? BETWEEN C1 AND C2
Non-Indexable

• SELECT ....FROM ...WHERE C1 <= ? AND C2 >=?
Indexable

17

• Keep the last 5 versions

SELECT DISTINCT SUBSTR(SYSPACK.VERSION,1,46) AS VERSION,
DATE(TIMESTAMP) AS TARIH FROM SYSIBM.SYSPACKAGE SYSPACK WHERE
SYSPACK.COLLID LIKE 'CRB%‘ ORDER BY TARIH DESC;

+-------------------------------------------------------------+
! VERSION ! TARIH !
+-------------------------------------------------------------+
1_! isb50_9440fp5_DEV_100701 ! 2010-07-12 !
2_! isb50_9440fp4_DEV_100416 ! 2010-05-06 !
3_! isb50_9440fp3_DEV_100402 ! 2010-04-09 !
4_! isb50_9440fp3_DEV_100305 ! 2010-03-18 !
5_! isb50_9440fp3_DEV_100305 ! 2010-03-17 !
6_! isb45_8210fp10_Hotfix001_091109_zOS ! 2009-11-12 !
7_! isb45_8210fp10_Hotfix001_091023_WAS61 ! 2009-10-23 !
+-------------------------------------------------------------+ 19

• DISABLE 4 versions except with the highest CREATETIME

REBIND PACKAGE (CRBMIG0.*.(isb50_9440fp4_DEV_100416)) FLAG(E) -
DISABLE(BATCH,CICS,DB2CALL,DLIBATCH,IMSBMP,IMSMPP,REMOTE,
RRSAF) )

DSNT408I SQLCODE = -807, ERROR: ACCESS DENIED: PACKAGE PGM01
IS NOT ENABLED FOR ACCESS FROM BATCH

20


• Free the rest of the packages

FREE PACKAGE (CRBDEV0.*.(isb45_8210fp10_Hotfix001_091109_zOS)) )

21

IMS & DB2
IMS DB2 Co-existence
• All DB2s in Data Sharing z-OS

LPAR
LPAR
COBOL LPAR LPAR
Application
IMS DB DB2 DB2
SQLs
Data
Sharing

AIX
API
• Proven Architecture
JVM
• High Availability, Scalability, Flexibility with DB2
Data Sharing and IPPlex

• Additional SQL Workload for IMS LPAR

22

Cobol & Unicode
• CREATE TABLE .... CCSID UNICODE

• Unicode Support in Enterprise COBOL
• Compiler Option : NSYMBOL(NATIONAL)
BIND Option : ENCODING(UNICODE)

• DATE Datatype
77 W-TARIH-YYAG PIC N(10).
DISPLAY FUNCTION DISPLAY-OF (W-TARIH-YYAG,
1026).

23

Cobol & Unicode

• IDTYP_VALUE GRAPHIC(4)

• 77 V-IDTYP-MUSNO PIC N(4).
...
MOVE N'csid' TO V-IDTYP-MUSNO.
SELECT COUNT(*) FROM CRBFID0.IP_ID
WHERE IDTYP_VALUE = :V-IDTYP-MUSNO

24

Unicode & Unload
• UNLOAD TABLESPACE DMIGCUST.SIP
PUNCHDDN TEMP_2 UNLDDN TEMP_1
FROM TABLE CRBMIG0.IP WHEN
(IP_USER_ID<>'DMUSER')

• IP_USER_ID GRAPHIC(6)

• IP_USER_ID<>x'444D55534552‘

• MR1030081910  Not working for V8, V9, V10

25

Unicode & Load
• All tables have GRAPHIC • Lot’s of external data in
datatype such as; Delimeted Format such as;

IFS_IDFR DECIMAL(15, 0) 11.,”9310”,2010-01-01
CARDT_VALUE GRAPHIC(4) 22.,”9343”,2010-01-02
CARD_EXPR_DATE

LOAD DATA INDDN CXZ837AQ LOG NO REPLACE
EBCDIC CCSID(01026,01208,01200)
GRAPHIC (4)  CHAR(4)
FORMAT DELIMITED INTO TABLE
ÜCRBMIG0Ü. ÜFNRATLÜ
( ÜIFS_IDFRÜ POSITION(*) DECIMAL
, ÜCARDT_VALUEÜ POSITION(*) CHAR(004)
, ÜCARD_EXPR_DATEÜ POSITION(*) DATE
EXTERNAL )
26

IDTHTOIN

• IDTHTOIN : Idle Thread Timeout in seconds
IDTHTOIN = 0 ; Disables Timeout!

• IDTHTOIN = 900 or 3600 or else... 

• MR1122081733 : Different IDTHTOIN settings for different
cases!

• DB2 V10 : Profile Tables 

27

IDTHTOIN
• DB2 V10 GA 

• For monitoring threads and connections

• SYSIBM.DSN_PROFILE_TABLE
PROFILEID : unique identifier for the profile
AUTHID, PLANNAME, COLLID, PKGNAME, IPADDR : Monitored
Items

• SYSIBM.DSN_PROFILE_ATTRIBUTES
PROFILEID : unique identifier for the profile
KEYWORDS, ATTRIBUTE1, ATTRIBUTE2 =
MONITOR IDLE THREADS, WARNING or EXCEPTION, threshold
value 28

INCORROUT
• SELECT *
FROM CRBFID0.ISB_PROVISION_INPUT T1
WHERE T1.PRV_IDFR = 134000001420080
AND T1.PRVIN_HOST_TS = (SELECT MAX (PRVIN_HOST_TS)
FROM CRBFID0.ISB_PROVISION_INPUT
WHERE PRV_IDFR = T1.PRV_IDFR
AND PRVIT_TYPE = T1.PRVIT_TYPE)

• SELECT * FROM CRBFID0.ISB_PROVISION_INPUT T1, ( SELECT MAX (PRVIN_HOST_TS) PRVIN_HOST_TS, PRVIT_TYPE
FROM CRBFID0.ISB_PROVISION_INPUT WHERE PRV_IDFR = 134000001420080
GROUP BY PRVIT_TYPE ) AS T2 WHERE T1.PRV_IDFR = 134000001420080
AND T1.PRVIN_HOST_TS = T2.PRVIN_HOST_TS

• Tested with DB2 for z V8, V9, DB2 for LUW & Oracle

• PMR - 20549,001,862 = Apar Fix: AM05402

29

Tuning Experiences
CLASS 2 TIME DISTRIBUTION
-------------------------------------
CPU !==> 4%
SECPU !
NOTACC !========================> 48%
SUSP !========================> 48%  Accounting Reports
AVERAGE APPL(CL.1) DB2 (CL.2)
------------ ---------- ----------
ELAPSED TIME 2:06:13.02 1:46:04.64

SQL and Index Design
Buffer Pools

Physical Design, Partitioning...
30

Tuning Experiences
DASD AVG RESP TIME : .927

CPU Activity:

LPAR Busy: 40.04  RMF Reports
MVS BUsy: 100.0

Distribution of address spaces in IN
READY QUEUE <=N: 3.2 %

Physical CPU was “saturated”
Little Paging as well.

31

Tuning Experiences

AVERAGE APPL(CL.1) DB2 (CL.2)
------------
ELAPSED TIME
----------
20:13.1970
----------
17:41.2783
 Accounting Reports

Almost %500 Better Performance !

32

Tuning Experiences
Batch Cobol
Will Update 100M Records with nice Commit freq. 1-2
Million Updates in 15-20 seconds
Then slows down significiantly
+ Waits Count Total
+ --------------------------------- ---------- ------------
+ Synchronous I/O Wait 19834 00:00:21.688
+ Asynchronous Read I/O Wait 34 00:00:00.074
+ Log Write I/O Wait 360252 00:07:48.482
+ Sync EX Unit Sw-com/abort/dealloc 22 00:00:00.032
+ ------------------------------------------------------------
+ Total Class 3 Wait Time 00:08:27.970

Check RMF Reports!
CPU Cap after 10-20 sec.
With enough CPU  40,000 Update/sec
33

ODS Application Migration
ODS Application

Near 7/24 Application
Business Logic : Cobol Stored Procedure

Data Migration  DRDA & XLoader & Parallelism
34

ODS Application Migration
DSNL030I -PR4B DSNLILNR DDF PROCESSING FAILURE FOR
LUWID=GA521E67.E4ED.000000000000
AUTHID=-N/A-, REASON=00D31034

The MAX REMOTE CONNECTED limit was reached

ETL Jobs
Feeding ODS from OLTP, DW and more!...

Idle Thread  3600 !

CondBat (max remote connected) is increased!
35

V9 Migration Update
RSU0908 Applied to V8
Migration/fallback : II14401, II14464, II14441
Development and Prod DB2 in Sandbox Lpar
V9 Order Arrived
V9 CM Install to Sandbox Lpar
Fallback, Remigrate, ENFM, NFM
Test CM, Prod CM, Test NFM, Prod NFM

36
36

V9 Migration Gotchas!
Before Plan Stability usage: SPT01 is enlarged 3
times more
GRAPHIC was seen as VARGRAPHIC and vice versa
in IBM Admin Tool 7.2.0
APAR PM17236  PMR 23637,001,862
DSNJ111E -AT3B OUT OF SPACE IN ACTIVE LOG DATA SETS
DSNJ115I -AT3B OFFLOAD FAILED, COULD NOT ALLOCATE AN ARCHIVE DATA SET
AT3BMSTR(REGION=32K  REGION=0M)
DB2 Crash during OMPE Start  Addresed in
PK68377
37
37

Thinking V10
GA Since Oct 22nd
DB2 10 is optimized on zEnterprise
CPU savings … right out of the box
zEnterprise Upgrade to
Hardware DB2 10

100 19% less CPU usage
80

60

40

20

0

DB2 9.1 DB2 10 38

Thinking V10

• Performance improvement Categories

1. Improvements that you receive by simply migrating to
Version 10
2. Improvements that are implemented by DBA-level
changes, without requiring application changes
3. Improvements that require application changes
4. Improvements from scalability enhancements

39

Thinking V10

1. Improvements that you receive by simply migrating to V10

• Out-of-the box savings in NFM
• Improved index matching for OR and IN predicates
• Parallelism enhancements such as for multi-row fetch
• Collection of autonomic statistics. Stored procedures will
determine if statistics need to be collected then DB2
performs the collection.

40

Thinking V10

2. Improvements that are implemented by DBA-level
changes, without requiring application changes

• New Access Path: Hash Access
WHO can get benefit ?
1-Queries uses equal predicates to access a single row
2-Tables of a predictable and reasonably static size.

41

Thinking V10

3. Improvements that require application changes

• Access to currently committed data it minimizes
transaction suspension
HOW ?
1- a new bind option
CONCURRENTACCESSRESOLUTION
2- two new PREPARE statement clauses,
USE CURRENTLY COMMITTED and WAIT FOR
OUTCOME, which you can specify in the attribute-string.
42

Thinking V10

4. Improvements from scalability enhancements

• The scalability enhancements contributes to improved performance

Reductions in log latch contention

Most cases, DB2 holds the log latch for less time when creating log
records, because of the increased use of compare and swap logic.

Enhancement does NOT require configuration or application changes.
Provides reductions in transaction and CPU time when logging rates
are high.
43

Thinking V10

MORE...

• 10x more concurrent users, up to 20,000 / subsystem
(MAXDBAT zparm)
• Temporal Data
• Thread Management thru Profile Tables

44

Cuneyt Goksu
VBT
IBM DB2 Gold Consultant
cuneyt.goksu@vbt.com.tr
F13
Lessons Learned from Isbank – A Story of a DB2 for
z/OS initiative

45

Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative

Recommended

More Related Content

What's hot (20)

Similar to Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative (20)

More from Cuneyt Goksu (20)

Recently uploaded (20)

Lessons learned from Isbank - A Story of a DB2 for z/OS Initiative