Scribd
Upload
4 views

Note 2074981 - How to Record Itab Leak Trace

SAP Note 2074981 provides guidance on how to trace and identify memory leaks related to internal tables (itab) in SAP HANA. It outlines symptoms, reasons for memory retention, and detailed steps for enabling tracing, analyzing memory consumption, and reducing memory usage. The note emphasizes that tracing should be conducted under the guidance of SAP HANA Development Support, especially in production environments.

Uploaded by

Mitchel Neyra
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views

Note 2074981 - How to Record Itab Leak Trace

SAP Note 2074981 provides guidance on how to trace and identify memory leaks related to internal tables (itab) in SAP HANA. It outlines symptoms, reasons for memory retention, and detailed steps for enabling tracing, analyzing memory consumption, and reducing memory usage. The note emphasizes that tracing should be conducted under the guidance of SAP HANA Development Support, especially in production environments.

Uploaded by

Mitchel Neyra
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

SAP Note

2074981 - How to Record Itab Leak Trace


Component: HAN-DB-ENG (SAP HANA > SAP HANA Database > SAP HANA DB Engines), Version: 19,
Released On: 15.05.2020

Symptom
You observe gradual increase of heap memory being used by at least one of the following allocators:
Pool/itab
Pool/itab/VectorColumn
Pool/JoinEvaluator/JERequestedAttributes/Results
Pool/parallel/aggregates
Pool/parallel/compactcol
The memory does not seem to be released over a significant time period.

Other Terms
leak i-tab internal table

Reason and Prerequisites


Memory pressure and retention on above allocators can occur for a variety of reasons. This can be considered normal
behavior as long as the internal tables, and subsequently memory pressure on them are released over time.
Some examples of memory consumed by these allocators that is considered normal behaviour:
Garbage collectors such as row store version consolidation will take care of created itabs within its cleaning cycle.
The completion of a transaction, which should also trigger itab cleaning.
Result caches, and natural eviction of compiled plans from plan cache.
Therefore, a high memory pressure and retention does not necessarily imply a memory leak.
Also, high peaks of these allocators can indicate poor application transaction design, and also do not imply a leak. For
example, long running transactions with queries consuming large hierarchies without filtering.
When memory consumption increases and does not seem to get released over a long period of time (such as many days) then
the tracing approach in this note can help identify areas that do not properly clear created itabs.
This note outlines initial, non-exhaustive tracing that can be used for identifying memory retention with the allocators
mentioned in Symptom section.

Solution
This trace method approach should only be done under guidance of SAP HANA Development Support in production
environments.
The following steps are used to collect the necessary traces:
1. With the database in a low memory pressure state (for example, after restarting or clearing the sql plan cache), take note
of the current time and enable the itab trace leak:

SELECT CURRENT_TIMESTAMP as ITAB_START_TIME from dummy;

ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') SET


('itab', 'trace_leaks') = 'yes',
('itab', 'trace_leakcallstack') = 'yes',
('itab', 'trace_addrefcallstack') = 'no',
('itab', 'leaktrace_max_callstacks') = '20000'
with reconfigure;

Important: Enabling this trace may have a performance impact, proceed with caution, avoid running this in
production unless instructed by HANA Development Support.
In older revisions (SP9 and lower) the database may have slow response when trace_leakcallstack is enabled.
In such cases, you may attempt to cancel the hanging thread by using the ALTER SYSTEM CANCEL SESSION
<connection_id>. The cancellation itself can also take several minutes.
If you're asked to enable 'trace_addrefcallstack' which rarely happens please double check with HANA Development
Support as it may cause indexserver crashed at "TRexCommonObjects::ItabLeakTraceWrapper::relRefImpl" after you
try to cancel a query while itab leak trace was being collected (SAP Note 2868606).

2. [Optional] Clear SQL Plan Cache immediately after activating itab leak tracing:

ALTER SYSTEM CLEAR SQL PLAN CACHE;


This step is only necessary if the memory pressure is already high as clearing the sql plan cache can help reducing itab
memory consumption.

3. [Optional] Preserve the current state of M_TEMPORARY_TABLES before the itab memory consumption increases:

CREATE COLUMN TABLE TEMPORARY_TABLES_BEFORE_MEM_GROWTH AS (SELECT * FROM


M_TEMPORARY_TABLES);

4. Wait for memory consumption to noticeably increase and retain (without decreasing). The memory pressure should be
representative, and depending on the scenario for which the trace is being activated, it may need to be active for a couple
of days or more. Use the query below or other variations, such as query HANA_Memory_TopConsumers_History from
SAP Note 1969700, to analyze the patterns of memory consumption to decide when to stop tracing:

SELECT * FROM
(SELECT TO_VARCHAR(CURRENT_TIMESTAMP) AS THETIME, HOST,PORT, CATEGORY,
ROUND((EXCLUSIVE_SIZE_IN_USE) / 1024 / 1024, 2) AS USED_MEMORY_SIZE_IN_MB
FROM M_HEAP_MEMORY
WHERE CATEGORY IN ('<allocator>')
ORDER BY HOST,PORT,USED_MEMORY_SIZE_IN_MB DESC)
UNION ALL
(SELECT THETIME, HOST, PORT, CATEGORY, MAX(USED_MEMORY_SIZE_IN_MB) AS
USED_MEMORY_SIZE_IN_MB FROM
(SELECT TO_VARCHAR(SERVER_TIMESTAMP) AS THETIME, HOST, PORT, CATEGORY,
ROUND((EXCLUSIVE_SIZE_IN_USE) / 1024 / 1024, 2) AS USED_MEMORY_SIZE_IN_MB FROM
"_SYS_STATISTICS"."HOST_HEAP_ALLOCATORS"
WHERE CATEGORY IN ('<allocator>'))
GROUP BY HOST, PORT, CATEGORY, THETIME)
ORDER BY THETIME DESC;

5. Record temp tables and itab leak traces in separate persistent tables:

CREATE COLUMN TABLE TEMPORARY_TABLES_AFTER_MEM_GROWTH AS (SELECT * FROM


M_TEMPORARY_TABLES);
CREATE COLUMN TABLE ITAB_LEAKS_AFTER_MEM_GROWTH AS (SELECT * FROM SYS.M_DEV_ITAB_LEAK
ORDER BY CREATION_TIME ASC);

6. Attempt to reduce memory utilization. This is normally done by clearing the SQL Plan Cache, as many of the itabs are
related to entries in the SQL Plan cache or the cache itself. In order to do that, use the commands below:

ALTER SYSTEM CLEAR SQL PLAN CACHE;


ALTER SYSTEM CLEAR COLUMN RESULT CACHE;

The memory usage might not reduce significantly with these commands.

In case there is significant reduction, there is a good chance the memory consumption is not leaking and following SAP
Note 2620830 might be more beneficial.

For other ways to attempt reducing memory consumption by allocators mentioned in Symptom see section -
Alternative Methods for Reducing memory pressure.

7. Record temp tables and itab leak traces in separate persistent tables again, to keep track of the involved objects created
since the start of the trace:

CREATE COLUMN TABLE TEMPORARY_TABLES_AFTER_MEM_GROWTH_CACHE_CLEAR AS (SELECT * FROM


M_TEMPORARY_TABLES);
CREATE COLUMN TABLE ITAB_LEAKS_AFTER_MEM_GROWTH_CACHE_CLEAR AS (SELECT * FROM
SYS.M_DEV_ITAB_LEAK ORDER BY CREATION_TIME ASC);

8. Disable traces:
ALTER SYSTEM ALTER CONFIGURATION ('indexserver.ini', 'SYSTEM') UNSET
('itab', 'trace_leaks'),
('itab', 'trace_leakcallstack'),
('itab', 'trace_addrefcallstack'),
('itab', 'leaktrace_max_callstacks')
with reconfigure;

9. Wait for the Statistics server to collect fresh information from M_HEAP_MEMORY, by default this is done every 15
minutes. Create a copy of _SYS_STATISTICS.HOST_HEAP_ALLOCATORS_BASE based on the trace timeframe:

CREATE COLUMN TABLE HOST_HEAP_ALLOCATORS_BASE_COPY AS (SELECT * FROM


_SYS_STATISTICS.HOST_HEAP_ALLOCATORS_BASE WHERE server_timestamp between <time where the
itab_leak traces started from step 1 - ITAB_START_TIME> and current_timestamp );

10. Export the tables created in previous steps:

EXPORT
TEMPORARY_TABLES_BEFORE_MEM_GROWTH,
TEMPORARY_TABLES_AFTER_MEM_GROWTH,
ITAB_LEAKS_AFTER_MEM_GROWTH,
TEMPORARY_TABLES_AFTER_MEM_GROWTH_CACHE_CLEAR,
ITAB_LEAKS_AFTER_MEM_GROWTH_CACHE_CLEAR,
HOST_HEAP_ALLOCATORS_BASE_COPY AS BINARY INTO '/target/linux/folder' WITH THREADS 6;
Remarks: As it is costy to keep all call stacks for all itab leaks, parameter leaktrace_max_callstacks (set to 20000 on this
document) should have a sufficiently large number in order to capture the callstack associated with the suspicious statements.
If that's not available (e.g. call_stack column has value "creation callstack: <callstack omitted>" ) it might be necessary to
either (1) rule the statement out on another system and re-execute the itab_leak tracing so that the call stack is available; or
(2) restart the itab_leak trace with a larger value for leaktrace_max_callstacks. On the later option, take into account that this
column is LOB and could potentially influence LOB caching as well as reduce performance of
cloning SYS.M_DEV_ITAB_LEAK table.
Alternative Methods for Reducing memory pressure
Disclaimer: these methods should *not* be used without previous discussion with SAP HANA Development Support.
This is a non-exhaustive list of methods other than clearing SQL/result cache in order to attempt reducing itab memory
consumption:
Clear HANA caches on HANA 2 SP3+: a new way of clearing different existing caches - not including the SQL Plan
cache - exist: ALTER SYSTEM CLEAR CACHE
Shutdown applications connected to the database
Close Planning sessions and objects: only applicable for cases when HANA Planning is in use:
ALTER PLANNING SESSION "ADMIN" WITH PARAMETERS ( 'action'='close_all_sessions', 'object'='*',
'host'='<indexserver host>', 'port'='<indexserver service port>' );
ALTER PLANNING SESSION "ADMIN" WITH PARAMETERS ( 'action'='drop_runtime_object', 'object'='*',
'host'='<host>', 'port'='<port>' );

This document is referenced by


SAP Component Title
Note/KBA

3114746 Unexpected Memory Growth in Pool/JoinEvaluator/JERequestedAttributes/Results When Using


Temporary Table

You might also like