TFA - Trace File Analyzer Collector

09. – 12.12.2019
Frankfurt am Main
#ittage
TFA
Markus Flechtner
.. And the Oracle Support Tools Bundle
1

www.markusdba.net|.de
@markusdba
TFA
.. And the Oracle Support Tools Bundle
Markus Flechtner

Markus Flechtner
• Principal Consultant, Trivadis Germany GmbH, Düsseldorf
• Oracle since 1990: Development, Support, DBA
• Focus: RAC, HA, Upgrade + Migration
• Teacher: RAC, New Features, Multitenant,
PostgreSQL
• Co-Author of the book "Der Oracle DBA"
(Hanser, 2016)
@markusdba www.markusdba.net|.de

17.05.2022 TFA and the Support Tools Bundle
4

Agenda
• What's TFA?
• Availability & Installation – levels of confusion
• What can TFA do for you?
• And the other tools ..
• Summary & Further Information
6
Picture
from
Pixabay

What's TFA?
... and the Support Tools Bundle?
7

Real life experience ..
• 26 node cluster
• 5 databases
• Strange ASM issue
• Oracle Support requested
• Clusterware logs
• ASM alert.logs
• Database alert.logs
• OS logs
8
For all 26 nodes

Trace File Analyzer Collector: Originally,..
• Collects trace and log files and system information from all nodes into a cluster with a single
command initiated on one cluster node
• Initial release in January 2013
• Centralized output
• Real-time scanning for specific error messages possible  Automatic Collection
• When uploading TFA results (ZIP file) to Oracle Support, Oracle has got the "Trace File Analyzer"
available to analyze the issue
9

Evolution Part 1: "Support Tools Bundle"
• Over time, more and more standalone support tools were integrated in TFA
• TFACTL as the central CLI for
• Oswatcher
• Oratop
• Pstack
• Procwatcher
• […]
• Orachk/exachk
10

Now: "Autonomous Health Framework"
• With release 19.3.0 (October 2019) OraChk and TFA were merged into the
"Autonomous Health Framework" (AHF)
• AHF consists of
• OraChk
• ExaChk
• TFA (and the Support Tools Bundle)
• Current version 19.3.2 (December 2019)
• Download from MOS 2550798.1
• Ca. 350 MB, incl. JRE
• For Windows and HP-UX, currently only standalone TFA (19.2.3) is available: MOS 1513912.1
11

AHF: History and Confusion - 2015
Source:
Oracle
Presentation on
AHF
12

AHF: History and Confusion - 2019
Source: AHF Users Guide
https://docs.oracle.com/en/engineered-systems/health-diagnostics/autonomous-health-framework/index.html
13

AHF: History and Confusion - Future
• Oracle 20c AHF Service
• Browser based UI
• REST API
• GIMR Service
• TFA Repository
Source: Oracle Corporation / Presentation at DOAG Conference 2019 / Oracle Safe Harbour Statements apply
https://www.doag.org/formes/pubfiles/11857280/2019-DB-Mark_Scardina-Protecting_DBs_With_Oracle_19c_Autonomous_Health_Service-
Praesentation.pdf
14

Availability & Installation –
levels of confusion
16

• TFA is included in the Oracle Grid Infrastructure software since 11.2.0.4
• TFA is included in the RDBMS software since 12.2
oracle@training19c:~/ [rdbms19000] cd $ORACLE_HOME/suptools
oracle@training19c:/u00/app/oracle/product/19.0.0.0/suptools/ [rdbms19000]
ls -al
total 8
drwxr-xr-x. 5 oracle oinstall 45 Apr 17 2019 .
drwxr-xr-x. 72 oracle oinstall 4096 Nov 27 20:02 ..
drwxr-xr-x. 12 oracle oinstall 4096 Oct 20 20:25 orachk
drwxr-xr-x. 2 oracle oinstall 20 Apr 27 2019 oratop
drwxr-xr-x. 3 oracle oinstall 21 Apr 17 2019 tfa
oracle@training19c:/u00/app/oracle/product/19.0.0.0/suptools/ [rdbms19000]
TFA – included in OracleSoftware (1)
17

• When installing the RDBMS software you can decide if you want to install TFA during the
execution of root.sh
Now product-specific root actions will be performed.
Do you want to setup Oracle Trace File Analyzer (TFA) now ? yes|[no] :
yes
Installing Oracle Trace File Analyzer (TFA).
TFA – included in OracleSoftware (2)
18
 If you decide to install TFA Collector later:
• Set ORACLE_HOME
• Run $ORACLE_HOME/suptools/tfa/release/tfa_home/install/roottfa.sh as root
• TFA is not updated with the RUs

TFA – separate installation
• Additionally, TFA is available as part of the Autonomous Health Framework (AHF)
• Current version 19.3.2 (except for Windows and HP-UX)
• No other standalone downloads available anymore
• When installing AHF will detect a running TFA instance and will update it in the existing directory
• For new installations, TFA suggests /opt/oracle.ahf as the TFA_HOME directory
19
oracle@training19c:~/ cd /u00/app/oracle/stage/
oracle@training19c:/u00/app/oracle/stage/ unzip AHF-LINUX_v19.3.2.zip
[...]
oracle@training19c:/u00/app/oracle/stage/ su
Password:
root@training19c:/u00/app/oracle/stage/mkdir -p /opt/oracle.ahf/data
root@training19c:/u00/app/oracle/stage/ ./ahf_setup
[...]

• TFA is running as a daemon in the background
• On a cluster, TFA is running on each node
root@training19c:/u00/app/oracle/stage/ [rdbms19000] ps -ef |grep tfa
root 1352 1 0 08:39 ? 00:00:00 /bin/sh /etc/init.d/init.tfa
run >/dev/null 2>&1 </dev/null
root 2593 1 1 08:39 ? 00:00:18 /opt/oracle.ahf/jre/bin/java -
server -Xms32m -Xmx64m -Djava.awt.headless=true -Ddisable.checkForUpdate=true -
XX:HeapDumpPath=/opt/oracle.ahf/data/training19c/diag/tfa
oracle.rat.tfa.TFAMain /opt/oracle.ahf/tfa
TFA daemon
20

What can TFA do for you?
21

TFA – original functionality
• Collect all relevant log and trace files from an Oracle cluster
• On a cluster, data collection is initiated on a single node
• Data is collected from the remote nodes and stored on the initiating node
• TFA can monitor the alert.log files and start a collection automatically ("autocollect") in case of a
ORA-600, ORA-7445, ..
22

oracle@training19c:~/ /opt/oracle.ahf/tfa/bin/tfactl diagcollect
By default TFA will collect diagnostics for the last 12 hours.
For more targeted collections enter the time of the incident, otherwise hit <RETURN>
to collect for the last 12 hours
[YYYY-MM-DD HH24:MI:SS,<RETURN>=Collect for last 12 hours] :
Collecting data for the last 12 hours for all components...
Collecting data for all nodes
[..]
2019/12/02 09:20:57 CET : Collection Name : tfa_Mon_Dec_02_09_20_49_CET_2019.zip
2019/12/02 09:20:57 CET : Collecting diagnostics from hosts : [training19c]
2019/12/02 09:20:57 CET : Scanning of files for Collection in progress...
[..]
2019/12/02 09:21:30 CET : Completed collection of additional diagnostic information...
.------------------------------------.
| Collection Summary |
+-------------+--------+------+------+
| Host | Status | Size | Time |
+-------------+--------+------+------+
| training19c | | | |
'-------------+--------+------+------'
Logs are being collected to:
/opt/oracle.ahf/data/repository/collection_Mon_Dec_02_09_20_49_CET_2019_node_all
Tfactldiagcollect – Example (single node)
23

oracle@training19c:/opt/oracle.ahf/data/repository/collection_Thu_Nov_28_22_47_00_CET_
2019_node_all/ [rdbms19000] ls -al
total 444
drwx------. 2 oracle oinstall 4096 Nov 28 22:47 .
drwxr-xr-x. 5 root root 4096 Dec 2 09:21 ..
-rw-r--r--. 1 oracle oinstall 2294 Nov 28 22:47
diagcollect_20191128224659_training19c.log
diagcollect_console_20191128224659_training19c.log
training19c.tfa_Thu_Nov_28_22_47_00_CET_2019.zip
training19c.tfa_Thu_Nov_28_22_47_00_CET_2019.zip.txt
Tfactldiagcollect – Result
24

• Which data is collected by default?
• alert.log from all databases - Patch Information
• ASM log files - CHM information
• listener.log files - Clusterware logs
• Components, node list and time window can be specified
• Data is "trimmed" to the relevant time window
2016/05/03 10:36:56 CEST : Total Size of all Files Checked : 3GB
2016/05/03 10:36:56 CEST : Number of files containing required range : 271
2016/05/03 10:36:56 CEST : Total Size of Files: 175MB
2016/05/03 10:36:56 CEST : Number of files trimmed : 18
2016/05/03 10:36:56 CEST : Total Size of data prior to zip : 93MB
2016/05/03 10:36:56 CEST : Saved 112MB by trimming files
2016/05/03 10:36:56 CEST : Zip file size : 7MB
Tfactldiagcollect – compression
25

TFA – Autocollect
28
Source: Oracle AHF Users Guide

TFA – and more (1)
• Collect information for Service Requests
• The "srdc_profile" specifies the problem type
• Examples:
• ORA-00600,ORA-01031, ORA-01555, ORA-4031, ORA-07445 and other ORA-errors
• Dbasm
• Dataguard
• Dbperf
• Dbupgrade
• Dbpartition
30
tfactl diagcollect -srdc srdc_profile
[-tag tagname] [-z filename] [-last nh|d|-from time -to time | -for time]
-database database

TFA – and more (2)
• Upload diagnostic information to My Oracle Support
• Internet connection required, so in many cases not applicable for production systems
• Can be combined with a specific srdc_profile (see previous slide)
31
# speficy MOS username and password and store them in a wallet
tfactl setupmos
# run diagcollect for a service request
tfactl diagcollect […] -sr <sr_num>
# upload the collected data to MOS
tfactl upload -sr <sr_num> -wallet <list_of_files>

And the other tools …
32

Support Tools Bundle – Overview (1)
33
Category Tool Function
ADR management managelogs manages files in the ADR, e.g. automatically
purges files after some time
files in the ADR alertsummary checks all alert- files for ORA-messages
tail tail on alert.log or trace files
events reports events in all the alert.log files (e.g. start
of an instance)
grep greps all files in ADR for a pattern

Support Tools Bundle – Overview (2)
34
Category Tool Function
processes ps greps processlist
oratop like "top" on the OS level
oswatcher logs OS usage, can produce graphs
triage evaluates results of oswatcher
pstack Process stack for a given process
.. and more

• TFA can manage (purge) the files in the ADR
• Usage:
tfactl managelogs -purge -older <num>d
tfactl managelogs -purge -older <num>d -dryrun
ADR management – managelogs (1)
35

• Example:
• Additionally, TFA can manage the files in the ADR automatically
• Disabled by default
• Interval and maximum age of the files can be defined
root# /opt/oracle.ahf/bin/tfactl set manageLogsAutoPurge=ON
Successfully set manageLogsAutoPurge=ON
root# /opt/oracle.ahf/bin/tfactl set manageLogsAutoPurgeInterval=120
Successfully set manageLogsAutoPurgeInterval=120
root# /opt/oracle.ahf/bin/tfactl set manageLogsAutoPurgePolicyAge=34d
Successfully set manageLogsAutoPurgePolicyAge=34d
ADR management – managelogs (2)
36
Interval in minutes,
default=60
Max Age. Can be
specified in "h" and d"

• Example:
• Scans all alert.log files in the ADR for events and critical errors
• Usage:
tfactl alertsummary [-dg <dbname>]
oracle@training19c:~/ [TVDCDB1] /opt/oracle.ahf/bin/tfactl alertsummary
[..]
------------------------------------------------------------------------
27 11 2019 20:51:22 Ora-07445 jol_superintf_set()... TVDCDB1_ora_12216.trc
03 12 2019 16:14:15 Database started
Summary: Ora-600=0, Ora-7445=1, Ora-700=0
~~~~~~~
Warning: Only FATAL errors reported
Warning: These errors were seen and NOT reported
Ora-27366 Ora-12012 Ora-27369 Ora-27451 Ora-65054 Ora-19815 Ora-19809
Ora-19804 Ora-16038 Ora-00312 Ora-38706 Ora-38708
alertsummary
37

• Example:
• Runs the "tail" command on a given file in the ADR (wildcards allowed)
• Usage:
tfactl tail [-f] <filename>
oracle@training19c:~/ /opt/oracle.ahf/bin/tfactl tail -f alert_TVDNCDB*
Output from host : training19c
------------------------------
2019-12-03T16:13:13.362399+01:00
db_recovery_file_dest_size of 3000 MB is 4.14% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
tail
38

• Example
• "lists important events" on the system
• Basic Usage (more parameters available):
tfactl events –component asm|crs –database <db> -from <time> -to <time>
oracle@training19c:~/ [TVDCDB1] /opt/oracle.ahf/bin/tfactl events
------------------------------
Event Summary:
INFO :5
ERROR :0
WARNING :0
Event Timeline:
[Dec/02/2019 22:28:31.000]: [db.tvdncdb.TVDNCDB]: Starting ORACLE instance (normal) (OS id: 5005)
[Dec/03/2019 08:54:14.000]: [db.tvdcdb1.TVDCDB1]: Starting ORACLE instance (normal) (OS id: 30505)
[Dec/03/2019 16:14:11.000]: [db.tvdcdb1.TVDCDB1]: Starting ORACLE instance (normal) (OS id: 7858)
events
39

• Example:
• "greps" files in the ADR for a given pattern
• Usage:
tfactl grep <search_pattern> <filename_pattern>
oracle@training19c:~/ [TVDCDB1] /opt/oracle.ahf/bin/tfactl grep ORA- alert*
------------------------------
Searching 'ORA-' in alert*
Searching /u00/app/oracle/diag/rdbms/tvdncdb/TVDNCDB/trace/alert_TVDNCDB.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
1462:ORA-12012: error on auto execute of job "SYS"."ORA$AT_SQ_SQL_SW_1082"
1463:ORA-13767: End snapshot ID must be greater than begin snapshot ID.
[..]
Searching /u00/app/oracle/diag/rdbms/tvdcdb2/TVDCDB2/trace/alert_TVDCDB2.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
3784:ORA-65011 signalled during: alter pluggable database demopdb2 close...
3786:ORA-65011 signalled during: alter pluggable database demopdb1 close...
[..]
grep
40

• Example:
• Combines ps & grep, i.e. greps the process list according to a given pattern
• Usage:
tfactl ps <pattern>
oracle@training19c:~/ [TVDCDB1] /opt/oracle.ahf/bin/tfactl ps smon
------------------------------
oracle 7105 1 0 16:13 ? 00:00:00 ora_smon_TVDNCDB
oracle 7935 1 0 16:14 ? 00:00:00 ora_smon_TVDCDB1
oracle 9021 9004 5 16:16 pts/0 00:00:00
/opt/oracle.ahf/python/bin/python /opt/oracle.ahf/ahf/lib/ahfctl.egg ps smon
ps
41

• "Realtime" database monitoring
• Usage:
tfactl oratop –database <dbname>
oratop
42

• Logs OS usage (CPU, Network, I/O)
• Collection usually runs in the background
• Enabled when installing TFA/AHF
Oswatcher (1)
43
oracle@training19c:~/ [TVDCDB1] ps -ef |grep osw
oracle 2815 1 0 16:03 ? 00:00:00 /bin/sh ./OSWatcher.sh 30 48 NONE
/opt/oracle.ahf/data/repository/suptools/training19c/oswbb/oracle/archive
oracle 2991 2815 0 16:03 ? 00:00:00 /bin/sh ./OSWatcherFM.sh 48
/opt/oracle.ahf/data/repository/suptools/training19c/oswbb/oracle/archive
oracle 25885 5898 0 16:51 pts/0 00:00:00 grep --color=auto osw
30 seconds interval
Keep data for 48 hours

• Data can be analyzed using "oswbba" (X-GUI required)
java -jar ./tfa/ext/oswbb/oswbba.jar -i
/opt/oracle.ahf/data/repository/suptools/training19c/oswbb/oracle/archive/
Starting OSW Analyzer V8.3.0
[...]
Enter 1 to Display CPU Process Queue Graphs
Enter 2 to Display CPU Utilization Graphs
Enter 3 to Display CPU Other Graphs
Enter 4 to Display Memory Graphs
Enter 5 to Display Disk IO Graphs
Enter 61 to Display Individual OS Process I/O RPS Graphs
Enter 62 to Display Individual OS Process I/O WPS Graphs
Enter 63 to Display Individual OS Process Percent User CPU Graphs
Enter 64 to Display Individual OS Process Percent System CPU Graphs
Enter 65 to Display Individual OS Process Percent Total CPU (User + System) Graphs
Enter 66 to Display Individual OS Process Percent Memory Graphs
Oswatcher (2)
44

Oswatcher (3) - Graphs
45

• Monitors database and clusterware processes (very often in connection with oradebug or OS
tools)
• Collects stack traces, wait chains, lock information, current SQL
• Results can be uploaded to Oracle Support
• Example:
oracle@training19c: /opt/oracle.ahf/bin/tfactl prw start
Tue Dec 3 18:00:30 CET 2019: Starting Procwatcher as user oracle
[..]
Procwatcher files will be written to:
/opt/oracle.ahf/data/repository/suptools/prw/oracle
[..]
oracle@training19c /opt/oracle.ahf/bin/tfactl prw stop
[..]
Tue Dec 3 18:01:32 CET 2019: Stopping Procwatcher
Tue Dec 3 18:01:33 CET 2019: Procwatcher Stopped
procwatcher
46

• Example:
• Process stack for a given process
• Usage:
Tfactl pstack <pattern>
oracle@training19c: ps -ef |grep smon
oracle 7105 1 0 16:13 ? 00:00:00 ora_smon_TVDNCDB
oracle@training19c: /opt/oracle.ahf/bin/tfactl pstack 7105
------------------------------
# pstack output for pid : 7105
#0 0x00007fc12549e3ca in semtimedop () from /lib64/libc.so.6
#1 0x0000000012915895 in sskgpwwait ()
#2 0x0000000012911b6b in skgpwwait ()
[..]
pstack
47
Can be PID or part of process name

Summary &
Further Information
48

Summary
+ TFA/AHF is a collection of helpful tools
+ Probably not all tools are useful for you (some are cluster specific)
+ But maybe you have your own tools for similar tasks (all DBAs have the same problems  )
+ Integrating all the support tools in a single package is a good idea
+ logfile collection, stripping and compression
+ ADR management
+ even more beneficial when used on a cluster
- Versioning, packaging and installation procedures are very "dynamic"
There's much more in TFA than shown in this presentation 
49

Further Information
• MOS-Notes
• TFA Collector - TFA with Database Support Tools Bundle (Doc ID 1513912.1)
• Autonomous Health Framework (AHF) - Including TFA and ORAchk/EXAChk (Doc ID 2550798.1)
• oratop - Utility for Near Real-time Monitoring of Databases, RAC and Single Instance (Doc ID
1500864.1)
• Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes (Doc ID
459694.1)
• OSWatcher Analyzer User Guide (Doc ID 461053.1)
• Documentation
• https://docs.oracle.com/en/engineered-systems/health-diagnostics/autonomous-health-
framework/ (as PDF 512 pages)
• Websites/Blog-Posts
• https://oracle-base.com/articles/misc/trace-file-analyzer-tfa
• https://oracle-base.com/articles/misc/autonomous-health-framework-ahf
50

Questions & Answers
Markus Flechtner
markus.flechtner@trivadis.com
Phone +49 211 5866 64725
@markusdba www.markusdba.net|.de

Mirko Hotzy
• Solution Unit Manager
Carl Pfaffenberger
Account Manager - Sales
Weitere Fragen oder Interesse an Trivadis?
Gerne stehen wir für Sie im Foyer zur Verfügung!

TFA - Trace File Analyzer Collector

Recommended

More Related Content

Similar to TFA - Trace File Analyzer Collector (20)

More from Markus Flechtner (20)

Recently uploaded (20)

TFA - Trace File Analyzer Collector

Editor's Notes