Heap Dump Analysis

- Holiday readiness-2020

Xiaobai Wang
Forward-Looking Statement
Statement under the Private Securities Litigation Reform Act of 1995:
This presentation contains forward-looking statements about the company’s financial and operating results, which may include expected GAAP and non-GAAP financial and other operating and non-operating
results, including revenue, net income, diluted earnings per share, operating cash flow growth, operating margin improvement, expected revenue growth, expected current remaining performance obligation
growth, expected tax rates, the one-time accounting non-cash charge that was incurred in connection with the Salesforce.org combination; stock-based compensation expenses, amortization of purchased
intangibles, shares outstanding, market growth and sustainability goals. The achievement or success of the matters covered by such forward-looking statements involves risks, uncertainties and assumptions. If
any such risks or uncertainties materialize or if any of the assumptions prove incorrect, the company’s results could differ materially from the results expressed or implied by the forward-looking statements we
make.
The risks and uncertainties referred to above include -- but are not limited to -- risks associated with the effect of general economic and market conditions; the impact of geopolitical events; the impact of
foreign currency exchange rate and interest rate fluctuations on our results; our business strategy and our plan to build our business, including our strategy to be the leading provider of enterprise cloud
computing applications and platforms; the pace of change and innovation in enterprise cloud computing services; the seasonal nature of our sales cycles; the competitive nature of the market in which we
participate; our international expansion strategy; the demands on our personnel and infrastructure resulting from significant growth in our customer base and operations, including as a result of acquisitions; our
service performance and security, including the resources and costs required to avoid unanticipated downtime and prevent, detect and remediate potential security breaches; the expenses associated with new
data centers and third-party infrastructure providers; additional data center capacity; real estate and office facilities space; our operating results and cash flows; new services and product features, including any
efforts to expand our services beyond the CRM market; our strategy of acquiring or making investments in complementary businesses, joint ventures, services, technologies and intellectual property rights; the
performance and fair value of our investments in complementary businesses through our strategic investment portfolio; our ability to realize the benefits from strategic partnerships, joint ventures and
investments; the impact of future gains or losses from our strategic investment portfolio, including gains or losses from overall market conditions that may affect the publicly traded companies within the
company's strategic investment portfolio; our ability to execute our business plans; our ability to successfully integrate acquired businesses and technologies, including delays related to the integration of
Tableau due to regulatory review by the United Kingdom Competition and Markets Authority; our ability to continue to grow unearned revenue and remaining performance obligation; our ability to protect our
intellectual property rights; our ability to develop our brands; our reliance on third-party hardware, software and platform providers; our dependency on the development and maintenance of the infrastructure
of the Internet; the
effect of evolving domestic and foreign government regulations, including those related to the provision of services on the Internet, those related to accessing the Internet, and those addressing data privacy,
cross-border data transfers and import and export controls; the valuation of our deferred tax assets and the release of related valuation allowances; the potential availability of additional tax assets in the future;
the impact of new accounting pronouncements and tax laws; uncertainties affecting our ability to estimate our tax
rate; the impact of expensing stock options and other equity awards; the sufficiency of our capital resources; factors related to our outstanding debt, revolving credit facility, term loan and loan associated with
50 Fremont; compliance with our debt covenants and lease obligations; current and potential litigation involving us; and the impact of climate change.
Further information on these and other factors that could affect the company’s financial results is included in the reports on Forms 10-K, 10-Q and 8-K and in other filings it makes with the Securities and
Exchange Commission from time to time. These documents are available on the SEC Filings section of the Investor Information section of the company’s website at www.salesforce.com/investor.
Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements, except as required by law.
Agenda
1. JVM Memory Structure
2. Heap Overview
3. Heap Dump Overview
4. Heap Dump Considerations
5. How to Generate Heap Dump
6. Shallow Heap vs Retained Heap
7. Tools for Heap Dump Analysis
8. Heap Dump Analysis Demo
Heap Dumps Analysis - JVM Memory Structure
JVM memory is divided into multiple parts: Heap Memory,
Non-Heap Memory, and Other.

Heap memory - Stores all java objects

Non - Heap Memory - Stores per-class structures such as
runtime constant pool, field and method data, the code for
methods and constructors, as well as interned Strings.

Other - Stores the JVM code itself, JVM internal structures,

loaded profiler agent code, and data, etc.

Reference:
https://www.betsol.com/blog/java-memory-mana
gement-for-java-virtual-machine-jvm/
Heap Dumps Analysis - Heap Overview
● Heap is specific to each JVM on each blade
○ Objects Stored in app blade1-3 are not
necessarily the same as in app blade1-4
● Heap is cleared upon the restart of the JVM (app
server restart)
● Object Relational Mapping (ORM) is used to
create virtual objects from database entries.
ORM has its own cache in heap
● Java manages its own heap through a process
called garbage collection. When an object is no
longer referenced it is garbage collected, freeing
up the allocated memory.
Heap Dump Analysis - Heap Dump Overview

What is a Heap Dump?

A heap dump is a memory screenshot of a java process.
The snapshot contains information about the Java objects
and classes in the heap at the moment the snapshot is
triggered.

What to Look For?

● High garbage collection results in high thread usage, not
the other way around.
● If you see high GC and high thread usage, always grab a
heap dump
● High GC on single Blade is often job related
● High GC on multiple Blades is normally storefront related
● Try to get an idea of what’s happening on the realm first
○ Running Data/Code Replication?
○ Jobs?
○ Index Rebuild?
Heap Dump Analysis - Heap Dump Considerations

Heap Dump Considerations

● Don’t restart the JVM unless you have to do so
○ Restarting the JVM will terminate all jobs &
replications running on the JVM
○ Consider realm activity before generating a heap
dump
● High GC must be occurring when the dump is generated
○ Can be hard to time considering GC spikes up/down
○ Narrow down to 1 hour view
Heap Dumps Analysis - How to Generate
Steps to Generate Heap Dump
1. Click bug icon next to the blade
2. Always Select Escalate
a. Heap Dump is emailed to everyone
3. Restart Process: Don’t check it
4. Dump Type: Select Heap
5. Select Blade (Only select 1 blade)
a. Use Grafana hourly view to time heap dump
b. Look for the blade with peak GC or upward
trending
6. Click Run
7. Check Email
a. Heap Dumps take longer time to generate and
email
b. A corresponding thread dump is also generated
Heap Dumps Analysis - Shallow Heap Vs. Retained Heap
Shallow Heap - an object own size in the memory
Retained Heap - the amount of memory that will be
freed when a particular object is garbage collected.
Example: Assume each object has 10 bytes
Object A: SH=10b, RH=70b
Object B: SH=10b, RH=30b
Object C: SH=10b, RH=30b
Object D: SH=10b, RH=10b
Object E: SH=10b, RH=10b
Object F: SH=10b, RH=10b
Object G: SH=10b, RH=10b
Heap Dumps Analysis - Shallow Heap Vs. Retained Heap
(Continue)
What are retained heap size of object A and H?
Object A: SH=10b, RH=?b
Object H: SH=10b, RH=?b
What are retained heap size of the following objects?
Object B: SH=10b, RH=?b
Object C: SH=10b, RH=?b
Object D: SH=10b, RH=?b
Object E: SH=10b, RH=?b
Object F: SH=10b, RH=?b
Object G: SH=10b, RH=?b
Heap Dumps Analysis - Tools
Eclipse Memory Analyzer (https://www.eclipse.org/mat/downloads.php)
Yourkit Java Profiler (https://www.yourkit.com/java/profiler/)
…
Heap Dumps Analysis - Demo with Yourkit Java Profiler