PeopleSoft

Performance
Metrics
[email protected]
[email protected]
 Abbey key facts 1

 Sixth largest bank by assets in the UK

 Founded in 1944
 Currently have approximately 18m customers
 741 branches across UK

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Abbey key facts 2

 Abbey's main offices are in London, Milton Keynes,

Bradford, Glasgow and Belfast.
 We have around 26,000 people (full time
equivalent)
 We have about 1.8 million shareholders
 Assets at 30 June 2004 - £171 billion
 Personal Financial Services trading profit before
tax for 6 months to 30 June 2004 - £340 million

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 History of PeopleSoft at Abbey

 PeopleSoft HRMS acquired for recruitment in 1994

 Implemented PeopleSoft HRMS in 1997
 Recruitment, Personnel & Training
 Paylink used to send data from HRMS to payroll
 Workflow and self-service with v7.5 in 2000
 JAVA HTML Clients
 PeopleSoft HRMS upgraded to 8 SP1 in 2001
 Implemented PeopleSoft Payroll in August 2003
 Project initiated to upgrade to HCM 8.8

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Current Platform

 AppServer runs on SUN E4500

 Database runs on SUN E10000
 Both boxes are shared with other applications
 Tier 1 mirrored disks
 Oracle 9.2.04

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Why PeopleSoft Payroll ?

 Integrated HRMS
 Common infrastructure
 Web enabled
 Automate administrative functions
 Manager Self-Service
 Absence and maternity input
 Employee Self-Service
 Overtime input
 On-line payslips
 Real-time data input
 Increase system availability

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 PeopleSoft Payroll Implementation

 Development commenced in January 2002

 In-house IT Project team
 Project delays due to re-scoping and internal re-structure
 Streamed payroll during parallel run tests
 Went live with payroll and absence in August 2003
 30,000 staff and 7,000 pensioners
 12 streams introduced in February 2004
 Introduced hash partition in July 2004 due to increased run
times
 Identify and calculate taking 2.5 hours
 But we had to tune it…

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Agenda

 What is Performance Tuning?

 On-Line
 Batch

 What has been achieved at Abbey?

 PeopleSoft Performance Monitor

 The shape of things to come!

 Questions?

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Rules of Engagement

 If you can’t hear me - say so now!

 Feel free to ask on-topic questions during the
presentation.

 This presentation (with notes) will be available on

the Conference website on Customer Connection
and on
 www.go-faster.co.uk

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Who am I?

 Independent consultant  Book

 www.psftdba.com
 System Performance
Tuning
 Oracle Databases
 Unix
 Tuxedo
 PeopleSoft Apps

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 What is performance?

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Aphorism

 Performance is exactly what the user

perceives it to be. No more, no less.

 Poor performance is when the user’s

perception does not match their expectation.

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Aphorism

 Performance Tuning is a search for lost time.

 See  Optimizing Oracle

 ‘Yet Another Performance
Performance Profiling  Cary Millsap & Jeff Holt
Method’ (1999)
 by Anjo Kolk, Shari
Yamaguchi & Jim
Viscusi
 http://www.oraperf.com
/whitepapers.html

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Aphorism

 When you have eliminated the impossible,

whatever remains, however improbable, must be
the truth.
 A Study in Scarlet, Arthur Conan-Doyle

 It is a capital mistake to theorise before one

has data.
The Memoirs of Sherlock Holmes, Arthur Conan-Doyle

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Aphorism

 Detection is, or ought to be, an exact science. It

should be be treated in the same cold and
unemotional manner.
 The Sign of Four, Arthur Conan-Doyle

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 PIA (On-Line) Performance

 Architecture
 brief overview
 Sources of Metrics
 Graphical Techniques
 Tracing
 Techniques

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 It isn’t always the database!

 Understand architecture
 Web front end
 Network
 Application server
 Database
 What can you measure?

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 PeopleSoft Internet Architecture

 PIA is 4-tier architecture

 Each component can contribute to response time
 Need to measure each component

Tuxedo
Tuxedo
Java
Java DBMS
DBMS
Application
Application
http / Servlet
Servlet Tuxedo Server
Server SQL (application
https Message (application
(presentation data
data&&meta-
meta-
(presentation (application
logic) (application data
data
logic) logic)
logic)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics (1)

 Browser
 Proxy Server
 3rd Party software
 Local agent on PC
 PeopleSoft Ping
 From PT8.19, PT8.4
 Web Server
 Access Log
 Weblogic & Apache
 Websphere does not generate a log

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics (2)

 BEA/Tuxedo
 Service Trace
 TMADMIN
 LogFence 4
 Oracle Database
 SQL Trace (event 10046)
 PeopleTools SQL Trace
 Large overhead

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Hint

 All the data collected on a time base.

 Therefore, make sure that the clocks on all
machines are synchronised at least with each
other.
 PC clocks are particularly liable to drift.

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics

Proxy Server
Access Log

Webserver
Access Log

Tuxedo
Tuxedo
Java
Java Application
Application DBMS
DBMS
http / Servlet
Servlet Tuxedo Server
Server SQL
https Message (application
(presentation (applicationdata
data
(presentation &&meta-data
meta-data
logic) (application
(application
logic) logic)
logic)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Web/Proxy Access Log

 W3C Standard format for access log

 www.w3.org/pub/WWW/TR/logfile.html
 Used by Weblogic, IIS

 Apache is different

 WebSphere doesn’t have an access log!

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Web/Proxy Access Log

 Fields
 Date, Time
 Time went log entry emitted
 (End time not start time)
 Time Taken
 accurate to OS time units (1/100ths or 1/1000ths)
 only 1s on Apache
 Message size bytes
 Client IP addresses & DNS names
 URI stem & Query

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 How to Enable WebLogic 5.1 Log

#weblogic.properties
#required
weblogic.httpd.enableLogFile=true
weblogic.httpd.logFileName=access.log
weblogic.httpd.logFileFormat=extended
#recommended
weblogic.httpd.logRotationType=date
weblogic.httpd.logRotationPeriodMins=1440
weblogic.httpd.logRotationBeginTime=11-24-2000-00:00:00

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 How to Enable WebLogic 6.1 & 8.1 Log

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sample WebLogic Access Log

#Version: 1.0
#Fields: date time time-taken bytes c-ip c-dns cs-
method sc-status cs-uri-stem cs-uri-query
2002-03-15 21:51:03 0.11 2103 10.0.0.2 go-
faster-1 GET 200 /peoplesoft8/signin.css -
2002-03-15 21:51:04 0.05 1186 10.0.0.2 go-
faster-1 GET 200
/peoplesoft8/images/secondaryPSFTLogo.gif -
2002-03-15 21:51:10 3.414 8101 10.0.0.2 go-
faster-1 POST 200
/servlets/iclientservlet/peoplesoft8/
ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&Pa
nelGroupName=PERSONAL_DATA&RL=&target=main2

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Apache Access Log

#httpd.conf
LogFormat "%{%Y.%m.%d %H:%M:%S}t|%T|%B|%u|%h|%{User-
Agent}i|%>s|%m|%U|%q" monitoring
CustomLog logs/access.log monitoring

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sample Apache Access Log
2002.02.26 09:57:06|0|275|-|127.0.0.1|Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|
GET|/peoplesoft8/cache/PT_NEXTTAB_ENG_1.gif|

2002.02.26 09:57:17|0|31847|-|127.0.0.1|Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|
POST|/servlets/iclientservlet|?
ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&Pa
nelGroupName=JOB_DATA

2002.02.26 09:57:26|0|30959|-|127.0.0.1|Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 4.0; Q312461)|200|
POST|/servlets/iclientservlet|?
ICType=Panel&Menu=ADMINISTER_WORKFORCE_(GBL)&Market=GBL&Pa
nelGroupName=JOB_DATA

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 What does the access log tell us?

 A date and time for every request

 Who made the request?
 How long did it take to serve?
 What was requested?
 Which component (panel group)?

 Indicate possible bandwidth problems.

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Download Bandwidth
#version: 1.0
#Fields: date time time-taken bytes c-ip c-dns cs-method sc-
status cs-uri-stem cs-uri-query
2002-09-10 11:29:52 0.359 15832 141.93.249.74
141.93.249.74 GET 200
/XXXXXXXX/images/topPSFTLogo.gif -

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics

Tuxedo
Service
Trace

Tuxedo
Tuxedo
Java
Java Application
Application DBMS
DBMS
http / Servlet
Servlet Tuxedo Server
Server SQL
https Message (application
(presentation (applicationdata
data
(presentation &&meta-data
meta-data
logic) (application
(application
logic) logic)
logic)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 BEA/Tuxedo Service Trace

 Edit psappsrv.ubx file

 CLOPT=“-r -e APPQ.stderr…”
 -r enables trace to stderr file
 -e qualifies name of stderr file

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Edit psappsrv.cfg

 Add a new variable to the trace section

 Thus service trace can be enabled/disabled from
configuration file
 New variable will appear in the configuration
dialogue

[Trace]
TuxedoServiceTrace=-r

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Edit psappsrv.ubx

 Change every CLOPT entry in

psappsrv.ubx.
 File name is set manually.
 Same as queue name

RQADDR="APPQ"
CLOPT="{$Trace\TuxedoServiceTrace} -e {LOGDIR}{FS}
{MACH}_{$Domain Settings\Domain ID}_APPQ.stderr
{$PSAPPSRV\Spawn Server} -s@..{FS}psappsrv.lst -- -C
{CFGFILE} -D {$Domain Settings\Domain ID} -S
PSAPPSRV"

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Resultant psappsrv.ubb

 When domain is configured .ubb file is

generated. All variables are fully resolved.

CLOPT=”-r -e
D:\ps\hr88\appserv\hr88\LOGS\HR88_APPQ.stde
rr -p 1,600:3,10 -s@..\psappsrv.lst
-s@..\psqcksrv.lst -sICQuery
-sSqlQuery:SqlRequest -- -C psappsrv.cfg -D
HR88 -S PSAPPSRV"

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sample Tuxedo Service Trace

 Each service called to domain is logged

SERVICE PID SDATE STIME EDATE ETIME
------- --- ----- ----- ----- -----
@ICScript 390 1014717320 4586495 1014717322 4588397
@ICScript 390 1014717322 4588467 1014717322 4588537
@ICScript 390 1014717322 4588557 1014717323 4589259
@ICScript 390 1014717332 4598021 1014717332 4598171
@ICScript 390 1014717355 4621745 1014717361 4627554
@ICScript 390 1014717365 4631329 1014717366 4632511
@ICScript 390 1014717375 4641944 1014717377 4643306
@ICPanel 390 1014717380 4646761 1014717381 4647783
@ICPanel 390 1014717388 4654202 1014717391 4657567
@ICPanel 390 1014717396 4662614 1014717401 4667271

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Tuxedo Service Trace

 Service - Name of Service

 Pid - An OS PID of listener or handler
 Sdate/Edate - Start/End Date
 Seconds since 00:00hrs GMT 1st January 1970.
 Stime/Etime
 Start/End Time in OS time units (1/ 100ths or 1/1000ths)
 Hence calculate accurate service duration
 Time on Server - not queuing

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Txrpt Utility

SERVICE SUMMARY REPORT

SVCNAME 9a-10a TOTALS

Num/Avg Num/Avg
--------------- -------- -------
ICScript 8/1.43 8/1.43
ICPanel 6/3.47 6/3.47
JavaMgrGetObj 1/0.04 1/0.04
GetCertificate 1/0.73 1/0.73
--------------- ------- -------
TOTALS 16/2.06 16/2.06

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics

Tuxedo
tmadmin
script

Tuxedo
Tuxedo
Java
Java Application
Application DBMS
DBMS
http / Servlet
Servlet Tuxedo Server
Server SQL
https Message (application
(presentation (applicationdata
data
(presentation &&meta-data
meta-data
logic) (application
(application
logic) logic)
logic)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 BEA/Tuxedo TMADMIN Utility

 BEA Interactive command line interface

 Can be included in scripts
 -r for read only in monitoring scripts
 (this is a different –r)

 pq (printqueue)
 psr (printserver)
 pclt (printclient)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 PQ

 What queues in domain

 How many servers on queue
 How many requests queued
 (How much work on queue)
Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine
--------- ------------------- --------- -------- -------- -------
JSL.exe 00095.00200 1 - 0 - GO-FASTER+
JREPSVR.exe 00094.00250 1 - 0 - GO-FASTER+
PSSAMSRV.exe SAMQ 1 - 0 - GO-FASTER+
BBL.exe 54455 1 - 0 - GO-FASTER+
WSL.exe 00001.00020 1 - 0 - GO-FASTER+
PSAPPSRV.exe APPQ 1 - 0 - GO-FASTER+

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 PSR

 What servers exist?

 How many requests have they handled?
 What are they doing right now?

Prog Name Queue Name Grp Name ID RqDone Load Done Current Service
--------- ---------- -------- -- ------ --------- ---------------
BBL.exe 54455 GO-FAST+ 0 3 150 ( IDLE )
PSAPPSRV.exe APPQ APPSRV 1 9 450 ICPanel
PSSAMSRV.exe SAMQ APPSRV 100 0 0 ( IDLE )
WSL.exe 00001.00020 BASE 20 0 0 ( IDLE )
JSL.exe 00095.00200 JSLGRP 200 0 0 ( IDLE )
JREPSVR.exe 00094.00250 JREPGRP 250 5 250 ( IDLE )

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 PCLT

 Who is connected?
 Are they doing something?
 When did they connect?

LMID User Name Client Name Time Status Bgn/Cmmt/Abrt

--------------- --------------- --------------- -------- ------- -------------
GO-FASTER-1 NT WSH 0:21:14 IDLE 0/0/0
GO-FASTER-1 NT JSH 0:21:13 IDLE 0/0/0
GO-FASTER-1 NT tmadmin 0:00:00 IDLE 0/0/0
GO-FASTER-1 PS JavaClient 0:02:30 BUSY/W 0/0/0

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics

Oracle
SQL*Trace

Tuxedo
Tuxedo
Java
Java Application
Application DBMS
DBMS
http / Servlet
Servlet Tuxedo Server
Server SQL
https Message (application
(presentation (applicationdata
data
(presentation &&meta-data
meta-data
logic) (application
(application
logic) logic)
logic)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Oracle SQL Trace

 Tuning tool
 Session trace
 Format it with TKPROF
 (Or the www.HOTSOS.com profiler)
 Find long running SQL statements
 Total SQL Time for session
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 72289 173.06 175.45 7 14654 94 0
Execute 77140 64.02 67.37 205495 766343 170611 78630
Fetch 50599 33.20 34.28 17302 971624 80146 143218
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 200028 270.28 277.10 222804 1752621 250851 221848

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Small Application Servers

 Create small application with just a single

handler process
 Enable trace on that process
 Trace sample transaction (only one user)

sys.dbms_system.set_sql_trace_in_session(<sid>,<serial>,TRUE);

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics

Proxy Server Tuxedo Tuxedo Oracle

Access Log tmadmin Service SQL*Trace
script Trace
Webserver
Access Log

Tuxedo
Tuxedo
Java
Java Application
Application DBMS
DBMS
http / Servlet
Servlet Tuxedo Server
Server SQL
https Message (application
(presentation (applicationdata
data
(presentation &&meta-data
meta-data
logic) (application
(application
logic) logic)
logic)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Graphical Analysis

 Don’t drown in numbers

 Draw a picture
 Load Data into Database
 SQL_Loader (Oracle 9i External Table)
 Process Data
 Matching, Aggregation
 Oracle Analytic Function
 Load data into Excel
 Draw Graph

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sample SQL*Loader File
LOAD DATA
INFILE 'APPQ.stderr'
REPLACE
INTO TABLE txrpt
WHEN (1) = '@'
FIELDS TERMINATED BY WHITESPACE
TRAILING NULLCOLS
(service "substr(:service,2)" -- remove leading @
,pid
,stimestamp
":stimestamp/86400+2/24+TO_DATE('01011970','DDMMYYYY')"
,stime ":stime/100"
,queue ”’APPQ’"
,etime ":etime/100")

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Tips about Graphs

 Connect Excel directly to a database

 Aggregate data carefully
 Too little and excel draws it badly because it adds a black border
around an area
 Too much and you lose details
 Pivot tables are useful
 but Excel97 buggy
 Excel2000 prevents automatic x-y scatter
 Scatters and Trendlines
 joining dots can obscure data
 Excel supports 32000 points per series
 Linear fit trend-lines can mislead

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Example: CPU Utilisation

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Example: Servlet Times

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Longest http conversations

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Example: Tuxedo Service Time

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 tmadmin - PCLT - Connect Client Sessions

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 tmadmin - PCLT - Busy/Wait Users

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 tmadmin - PQ - Queue Length

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 tmadmin - PSR - print server processes

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Relating different metrics produces new information

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Query Monitoring in PT 8.4x

 Every time a query runs it updates average

execution times
 Can additionally log individual query executions for
specified queries.

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 The metrics are in the database

 Query the top 10 PS/Queries

SELECT * FROM (
SELECT RANK() OVER (ORDER BY tottime DESC ) as qryrank
, oprid, qryname, totexec, tottime
, 100*RATIO_TO_REPORT(tottime) OVER () as pcttime
FROM (SELECT oprid, qryname
, SUM(execcount) totexec
, SUM(execcount*avgexectime) tottime
FROM psqrystats
GROUP BY oprid, qryname) a
) WHERE qryrank <= 10
/
QRYRANK OPRID QRYNAME TOTEXEC TOTTIME PCTTIME
------- ---------- ------------------------------ ---------- ---------- -------
1 DMKPUB 4 2.8 51.3
2 PS DMK 7 2.4 43.6
3 PER701__DEPT_TBL 1 0.3 5.1

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Metrics for each query execution

 Top 10 Queries in the last 7 days.

SELECT * FROM (
SELECT RANK() OVER (ORDER BY tottime DESC ) as qryrank
, oprid, qryname, totexec, tottime
, 100*RATIO_TO_REPORT(tottime) OVER () as pcttime
FROM (SELECT oprid, qryname
, COUNT(*) totexec
, SUM(exectime) tottime
FROM psqryexeclog
WHERE execdttm > SYSDATE - 7
GROUP BY oprid, qryname) a
) WHERE qryrank <= 10
/
QRYRANK OPRID QRYNAME TOTEXEC TOTTIME PCTTIME
------- ---------- ------------------------------ ---------- ---------- -------
1 DMKPUB 4 2.8 51.3
2 PS DMK 7 2.4 43.6
3 PER701__DEPT_TBL 1 0.3 5.1

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Batch Performance

 Sources of Metrics
 Graph

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics (1)

 Process Scheduler Request Table

 PSPRCSRQST
 need to archive this table
 maintain history in another table with on DELETE trigger

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Graphical Analysis

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Sources of Metrics(2)

 Application Engine Steps

 PS_BAT_% tables
 AE_TRACE=1152=1024+128
 In Process Scheduler configuration (psprcs.cfg)
 Report to database and/or trace file
 Cobol Statement
 Report to trace file
 SQL Trace

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Top AE Statements

AE PC % Cum % % Cum %
Stmt Execute SQL SQL PC SQL SQL Total Total
Rank Statement ID Count Time Time Time Time Time Time Time
---- ---------------------------------------- -------- --------- -------- -------- ----- ------ ----- ------
1 GP_PMT_PREP.CA000.CF000.S 13 11655.56 0.00 0.00 12.8 12.8 19.2 19.2
2 PeopleCode.Rowset.Fill GP_RSLT_ACUM 4725754 10065.02 0.00 0.00 11.1 23.9 16.6 35.9
3 GP_PMT_PREP.FA000.FH000.S 15 9828.60 0.00 0.00 10.8 34.8 16.2 52.1
4 GP_PMT_PREP.FA000.FC000.S 15 8556.60 0.00 0.00 9.4 44.2 14.1 66.2
5 GP_PMT_PREP.FA000.FG000.S 15 8553.51 0.00 0.00 9.4 53.6 14.1 80.3
6 GP_PMT_PREP.FA000.FB000.S 15 8322.51 0.00 0.00 9.2 62.8 13.7 94.1
7 PeopleCode.Rowset.Fill GP_RSLT_PIN 3177786 6906.78 0.00 0.00 7.6 70.4 11.4 105.5
8 PeopleCode.CNuK_ 4725754 4696.51 0.00 0.00 5.2 75.6 7.8 113.2
9 PeopleCode.CDfF0_PPNw 3409180 2392.33 0.00 0.00 2.6 78.2 3.9 117.2
…

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Oracle SQL*Trace & TKPROF

 Enable SQL Trace on batch process

 Process trace file with TKPROF
 report Top n statements
 sort by
 Parse/Fetch/Execute
 CPU/Elapsed Time
 statement timings/execution plan

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 36 0.02 0.01 0 0 0 0
Execute 36 55.09 86.80 0 2973787 3 1
Fetch 0 0.00 0.00 0 0 0 0
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 72 55.11 86.81 0 2973787 3 1

 Hotsos Profiler

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Triggers

 Process Scheduler Status  Enable Trace

Change  Archiving data during purge
 After Login routines
 Altering session parameters

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Process Scheduler

CREATE OR REPLACE trigger sysadm.set_trace

BEFORE UPDATE OF runstatus ON
sysadm.psprcsrqst
FOR EACH ROW WHEN
(new.runstatus = 7 AND old.runstatus != 7
AND new.prcstype IN('Application
Engine','COBOL SQL',
'SQR Process','SQR Report','SQR
Report For WF Delivery'))

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 After Logon Triggers
CREATE OR REPLACE TRIGGER sysadm.psqrysrv_session_trigger
AFTER LOGON ON sysadm.schema
DECLARE
l_program VARCHAR(64);
BEGIN
SELECT program INTO l_program FROM v$session
WHERE sid IN(SELECT sid FROM v$mystat WHERE rownum = 1);
IF substr(l_program,1,8) = 'PSQRYSRV' OR
instr(upper(l_program),'PSNVS') > 0 THEN
sys.dbms_session.set_sql_trace(true);
...

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 For example…

sys.dbms_session.set_sql_trace(true);
ALTER SESSION SET EVENTS '10053 trace name context
forever, level 1’;
ALTER SESSION SET EVENTS '10128 trace name context
forever, level 2’;
ALTER SESSION SET optimizer_mode=FIRST_ROWS;
ALTER SESSION SET optimizer_index_caching=100;
ALTER SESSION SET optimizer_index_cost_adj=1;
ALTER SESSION SET sort_area_size = 100000;
sys.dbms_application_info.set_module('nVision','psqrys
rv_session_trigger fired');

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Further sources of Information

 PSFT Customer Connection Red Papers

 8.4 PIA, Power Users

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Its worth getting up early tomorrow!

 Session 511  Session 917

 Wednesday 6th October  Wednesday 6th October
 8.30-9.30  8.30-9.30
 Configuring Global Payroll  PeopleTools Performance
for Optimal Performance Monitor 8.44
 Lots of interesting DBA
Technical Stuff

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Performance Monitor in PeopleTools 8.44
Web Application
Application
Browser Web
Browser Server Server
Server
Server DBMS
DBMS
PIA APPQ PSAPPSRV
http / Servlet Tuxedo
SQL
https Message (application
(application
PPMI PSPPMSRV
Servlet
data
data&&meta-
meta-
(presentation
(presentation data
data
&&JavaScript)
JavaScript) Monitor (application
Servlet (application
logic)
logic)
Monitoring
System
Web
Web Application
Application
Browser
Browser Server
Server Server
Server
APPQ PSAPPSRV
DBMS
DBMS
Screen Java http / PIA Tuxedo
Paint Script Servlet SQL (application
https Message (application
PSMONITORSRV data
data&&meta-
meta-
data
data
(presentation
(presentation
&&JavaScript) (presentation (application
(application
JavaScript) (presentation
logic) logic)
logic)
logic)

Monitored System
PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.
For Internal Use Only, Do not distribute outside of PeopleSoft.
 View a PIA Transaction

 PMU – Performance
Monitoring Unit

 If you use Oracle

 SQL Trace with Wait
Events
 Event 10046,8

 Time interval Interface

for the PIA

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Application Server Instrumentation

 Each Application Server collects and sends metrics

to the monitoring servlet.

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Its also collects instantaneous metrics!

 Events
 JVM Status
 Web Site Status
 Web Servlet Status
 Resources per Process
 Host Resources
 Tuxedo pq/psr
 Master Scheduler Status/Detail
 PSPING

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 This will Revolutionise Performance Tuning
Browser/ Web Application Database
Network Server Server Time Time
Time Time
Tuxedo
Web Server Service PeopleTools Oracle
Access Log Trace Trace SQL*Trace

Web
Web Application
Application
Browser
Browser Server
Server Server
Server DBMS
DBMS
Screen Java http / Servlet Tuxedo APPQ
Paint Script Thread
PSAPPSRV SQL (application
https Message (application
data
data&&meta-
meta-
(presentation data
data
(presentation (presentation (application
(application
&&JavaScript)
JavaScript) (presentation logic)
logic)
logic) logic)

101: PIA Request 400: Tuxedo Service 400: PeopleCode only

115: Jolt Time 401: ICPanel
PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.
For Internal Use Only, Do not distribute outside of PeopleSoft.
 Aphorism

 The problems change

from release to release,
but the methods by
which they are
investigated remain the
same.
 Jonathan Lewis (author of
Practical Oracle 8i)

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Questions?

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
 Aphorism

 You know my methods, apply them.

 A Study in Scarlet, Arthur Conan-Doyle

PeopleSoft Proprietary and Confidential, Copyright 2004 PeopleSoft, Inc.

For Internal Use Only, Do not distribute outside of PeopleSoft.
PeopleSoft
Performance
Metrics
[email protected]
[email protected]