Memory and C++

1
debugging at Electronic Arts
By Scott Wardle
2 Introduction

 Memory interfaces and debug tools for C++ games

2000 (PS2) embedded C style
2005 (Xbox 360, PS3) Interface programming,
EASTL
Now (PS4, Xbox One) 64 bit address spaces
 Our Current Tools:
How all of our debugging systems work together
3 About me

Scott Wardle,
20+ years Game Dev
Solving problems through visualization and
drawing pictures
I am also badly dyslexic, so please note spelling
mistakes and inform me later after the
presentation 
4 Vocabulary

 Allocators, Arena, Heaps

Allocators (an object or interface that can alloc
and free)
Arena (a set of address ranges controlled by one
allocator)
From Arena find an Allocator
From Allocator find an Arena
Heap ~= Allocator + Arena
5 C style 2000s
Overview
Year ~2000: PS2 32M ram
Most people are using C++ compilers
STL is not used
No virtual memory
Nearly no OS
Similar to embedded systems
6 C style 2000s
Interfaces for speed
 Macro per class
#define NEW_DELETE_OPERATORS(debug_name)
Good for fixed sized pools or slabs of objects
class CollisionChooser {
public:
NEW_DELETE_OPERATORS(CollisionChooser)
…
};
7 C style 2000s
Interfaces for debug
 Global new
 void* operator new(size_t size, const char* debug_name,
int flags=MB_LOW)
Debug name and
Header Allocated Block footer sentinel stored in
footer
Header Allocated Block footer
Note debug_name split into
Header Allocated Block “category::alloc” example:
footer
“render::player”,
“gameplay::physicsmesh”
 8 C style 2000s
Allocation technology
Almost all memory is in one heap
Well we did have a simple small block allocator
We had to work hard at defragmentation
Low Memory High Memory
Decompress
Small Load
Alloc General
Block compressed Texture 0
Mesh
Texture 0 Allocator
Allocator
9 2005 Overview
2004 - Xbox 360, PS3 (512M ram)
Virtual memory! - NO HDD , No GPU support,
32 bit
All consoles have multiple CPUs
 (Not just for Sega Saturn)

The main changes for 2005:

Support for multiple allocators
Better tracking and logging tools
Stomp allocator!!
Memory tracking with EASTL
10 2005 Support for Multiple Allocator

Polymorphic Allocator

SQLQuery *NewQuery(ICoreAllocator* a) {
return CORE_NEW(a, "sql", MEM_LOW) SQLQuery(a);
}

void DeleteQuery(ICoreAllocator* a, SQLQuery *sql){

CORE_DELETE(a, sql);
}

Calls ~SQLQuery()
not delete!!
 11 2005 Organizing Heaps/Arenas

Sub
Static Global Level Temp Time
Level

Medium
Small Size
Large

Gameplay Gameplay Render Render UI UI

Team
SBA Heap SBA Heap SBA Heap
A mix of time and size gives good defragmentation properties.
Organizing by team fragments heaps but easy to set blame.
So for my team we use all of these to varying degrees.
 12 2005 Team Based Heaps/Arenas vs
Team Based Categories
Memory Corruption between teams sucks

R S Medium
Small Render 0 SIM0 SIM1 Render 2 SIM3 Render 1
3 2 Large
Categories are a Fragmentation between
way to tags teams is hard. Who to
allocations so you blame when you are
can budget them out of memory?
together.
 2005 Better Tracking and Logging
13

Only sentinel Tracking live allocations in a Memory

stored in footer separate heap. Logging or
tracing system
Normal Heap Debug Heap
Allocated Category::
H F Address Size Alloc
Block Alloc Name
Memory
Allocated Category::
H F Address Size Alloc Logging
Block Alloc Name
To Disk
Allocated Category:: Free
Alloc
H F Address Size
Block Alloc Name
Hash Key
 14 2005 Logging

Start End of
of time time
Select Time

Category/ Alloc Alloc Alloc

Heap Name Count Size

whole
delta
snapshot
between
of memory
2 times
15 2005 Arena Block View

Green Systems
Yellow selected block
Purple Presentation
Grey Free

Info about
selected
block
 16 2005 Stomp Allocator!!
Stomp Allocator – so good it is worth it’s own
slide
Lots of memory, 4k per alloc

Crash!

Page 4KiB
Page 4KiB Alloc
Read Only
Read/Write
512 bytes
(Or not mapped)
No Crash but bad
Use sentinel? Or Flip
 2005 Ref Counted Pointers
17

Add a debug system for ref counts is hard:

A Tracking system would be like garbage collector…
A Logging system would generate even more data…

shared_ptr
Sim Render are useful !!
Oh No Player Player
memory but use
leak! unique_ptr
Collision particle or bare pointers
Mesh system for easy life times
18 2005 EASTL
A 2010 version of EASTL is available now from webkit.
Why EASTL
STL allocators are painful to work with
Intrusive containers, Ring Buffers, etc…
Superior readability and performance
Memory is Allocated in empty versions of some STL
objects
Etc…

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2271.html
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4526.pdf
 19 EASTL faster for optimized code

EASTL vs vc++ Dinkumw

EASTL is often a little optimized
EASTL
faster. In 71 out of
120

Faster
188 tests.
100

71
80

EASTL Even
NUM TESTS
• Faster means 1.3x
60

Slower 107
40

or better. 20
10
0

• Slower means 0.8x # Tests EASTL Slower # Tests Even

vc 2015
# Tests EASTL Faster

as quick or slower
20 EASTL MUCH faster for debug code

The same 188 tests EASTL vs vc++ Dinkumw EASTL

debug Faster
complied in debug 180
164
160

140

120

NUM TESTS
100

80

60 EASTL Even
40 Slower 19
20 2
0
# Tests EASTL Slower # Tests Even # Tests EASTL Faster

vc 2015 debug
 Open sourcing EASTL
21

EA is looking to open source EASTL

Roberto Parolin will be taking pull requests
Coming soon to:
https://github.com/electronicarts
Technical details announce later to SG14 group
 2005 EASTL Memory tracking problems
22

EASTL’s allocator were painful to track every object

You need to make a new type

typedef eastl::vector<int,EASTLICoreAllocator>
MyVec;

Then pass in a defaulted parameter

ICoreAllocator* alloc = GetGameplayAllocator();
MyVec vec(alloc);
 2005 EASTL Memory tracking problems
23

Default parameters at the end so hard to enforce use.

unordered_map ( size_type n = 1000
const hasher& hf = hasher(),
const key_equal& eql = key_equal(),
const allocator_type& alloc = allocator_type() );

Worked on all EASTL types but clumsy

EA::ICoreAllocator* alloc = GetRendAllocator();
vec.get_allocator().set_allocator(alloc);
 2005 EASTL Memory tracking problems
24

At first we hacked EASTL to make it easier

vector
v(eastl::allocator( "AI::Piano::Input" ))
But this meant our team couldn’t share code... with
other teams
(Accessing allocator by name was a bad idea
anyways)
 2005 EASTL Memory tracking problems
25

We also ran into type erasure problems

typedef vector<int, EASTLICoreAllocator> MyVec;

typedef vector<int> YourVec;
MyVec myVec;
YourVec yourVec;
myVec = yourVec; // what should happen here…
ERROR: no operator found which takes a right-hand operand
of type 'YourVec' (or there is no acceptable conversion)
 2005 “Good?” EASTL usage with EASTLICA
26

Wrap EASTL with EASTLICA to force usage of polymorphic

allocator
template <typename T>
class String : public base_string<T, EASTLICoreAllocator>{
String(ICoreAllocator *alloc, const char*name=“Str")
: basic_string<char, EASTLICoreAllocator>(
EASTLICoreAllocator( name, alloc ))
…
};
ICoreAllocator* alloc = GetStringAllocator();
EASTLICA::String str(alloc);
 2005 “Good?” EASTL usage with EASTLICA
27

Macro used to implement STL like types for each large system.
#define EASTLICA_VECTOR( EASTLICA_TYPE,
GET_DEFAULT_ALLOC, ALLOC_NAME )\
template< typename T> class EASTLICA_TYPE : public
EASTLICA::Vector<T>

Using Macro to create a STL-like types for a large system

EASTLICA_STRING( CareerModeString,
CareerMode::GetStringDefaultAllocator(), ”CareerStr" );
 2005 “Good?” EASTL usage with EASTLICA
28

This fixed our type type erasure problems.

CareerModeString str;
LocalizedString lstr = getStrId(42);
str = lstr; // woot no compile error! Both use same allocator.

This also fixed the ownership issues.

CareerMode owns its strings and localization does not own all
strings in the game.
Allocators are copied sometimes but not always.
 Today’s Memory System
29

PS4, Xbox One – Today 8GB (5GB for the game)

GPU memory does not have to be linearly mapped.
(GPU assets are still special case however.)
64 bit virtual address space and a HDD to swap to.

The big changes these days:

Debug Memory System
EASTL Memory Tracking
New debug tools
 Today’s Debug Memory System
30

Alloc debug names slowly die

void* operator new(size_t size, EA::ICoreAllocator* alloc)
The old interface exists. But uses scopes.
Scopes are everywhere
Resource and Asset Names
Alloc Name, Allocator, Category, and Call stacks
FB_MEMORYTRACKER_SCOPE(data->debugNames[i]);
FB_ALLOC_RES_SCOPE(data->debugNames[i]);
*(This does mean more thread local storage use)
 Today’s EASTL Memory Tracking
31

Everyone is still doing this:

class Team
{
int teamid;
eastl::vector<player> players;
}
Team*home = new (allocator) Team;

However EASTL is still a problem

 EASTL use parent arena by default tracking
32
Gameplay Arena
Team Home
(One Allocation) Don’t have to use
the same arena for
Check What int teamId; child
Arena parent vector<player> players;
is in allocator (0 bytes maybe) IE: use gameplay’s
first small block
last allocator
Allocate Child end not general
using parent allocator
Player 1
arena Player 2
as parameter Player 3
 EASTL use parent arena by default tracking
33
Problems
It does take some CPU time.
What about objects on the stack?
What about move operators?
Object in gameplay arena and move it to
rendering. Only the parent object will move.
“You made it you own it” logic works 80% of the
time.
For other cases use EASTLICA patterns.
(Systems that are factory for other systems.)
34 Today’s Debugging Tool DeltaViewer
History lesson over! Let’s look at today’s tools!

DeltaViewer displays a session of data.

A session is one run of the game
This data is sent from console to a http server on the SE’s or
QA’s computer
The data is stored in tables
These tables can be joined into views
35 DeltaViewer
Some popular views are:
TTY events debugging (Trace Log)
IO Load profiler (Turbo Tuner)
Frame rate and Job thread profiler (Performance Timer)
Memory Investigator, reviews memory leaks and
changes over time
Memory Categorization groups allocations at a given
time
36 TTY events debugging (Trace Log)

Level 1

Level 2
 37 IO Load profiler (Turbo Tuner)

Bundle is a group of
files that have to be
loaded to move the
Timeline
game to the next
level or sub level.
Bundles
Chunks are blocks of
data that are
steamed in. Like Chunks
movies or music or
terrain in open world
games.
38 IO Load profiler (Turbo Tuner)
Each Printf on the selected channel
gets an event line so you can undersand
when it happened
39 IO Load profiler (Turbo Tuner)

Loading Level 1 Playing Level 1 Loading Level 2 Playing Level 2

Why do I continue
to load bundle
while playing
40 IO Load profiler (Turbo Tuner)

Hover

Name of
the Bundle
41
Frame rate and Job thread profiler
(Performance Timer)
Each Rectangle is a Frame
The height is the time in ms
of this frame

Frames Expensive frame

Selected Frames in blue

Job Selected Frames

Show Up here
Expensive
Functions Calls Frame
From Job

Why wait for rendering

Start Frame End Frame

 42 Loading profiler + Frame rate profiler

We can combine views

Why?
Loading is about more then disk performance
Decompression
Stamping one texture on with a font
Recompressing and loading into VRAM
Loading is often limited by CPU
 Turbo
43 Tuner Add noisy screen from 4k screen
Selected
Frame time/frames
Rates
One BIG Frame Selected time/rames
CPU0

CPU1
View of
CPUs
and CPU2
Jobs
CPU3
CPU4
CPU5
GPU
 Use Memory Investigator for leaks
44

Finding memory leaks

How to find memory leaks in most games. Find:
A.Start of loading 1st level
B. End of loading 1st level
C.End of Loading 2nd level
(Growing objects look like leaks but often after a few levels
this goes away, wish we had realloc)
Capture allocs By C it should be
between A and B free
Alloc at Not Free at
T1 T2 LEAK!
 A B C
45
Capture Should
Allocs Be Free
Turbo
Tuner Here Before
Here

Different mode like growth

Diagram of the mode
or memory leaks
List of Ptr Call
leaks! Scope Asset
& Stack
BAD! Name
Size ID
Full Call stack of
Selected item
46
Memory Categorization
These times are often found Scrub to another
using turbo tuner time

Before After
 Memory Categorization
47

big allocs 2M or
greater take the
space

Lots of small allocs 512 bytes or smaller

 the code is 50MiB
Memory Categorization
small on this scale
48

• Rendering
(procedural
textures and
other buffers
used to draw
the scene)
• Content,
meshes,
textures and
entities that
tie these
together
49 Summary

DeltaViewer
Has many views:
TTY Event Timing
IO and Load times
Jobs and threads
Memory changes
We have a lot of work to do to ship the game I am on 
(Good thing I have one year left)
 50 Summary
EASTL and STL allocators
Hard to track
Use the “if you made it you own it rule”
Use the “this” pointer of allocator as a parameter for your
allocators
EASTLICA
Good at enforcing allocator use for a large group of SEs
Helped with type erasure problems in stl::string and other
classes. MyString does not work with YourString
51 Summary

Games in general
Most memory is used by large allocation
Most memory is mostly content (meshes and
textures) or rendering
There are a large number of small allocations.
Small block allocators, pool systems, slab allocators
are a good idea
Stomp Allocator are great (Use memory map to find
who stomped you…)
52 Questions?