NET Systems Programming Learned the Hard Way.pptx

.NET Systems Programming
Learned the Hard Way
By Aaron Stannard (@Aaronontheweb)
CEO, Petabridge
Creator, Akka.NET

About Me
• .NET developer since 2005 (college
internship)
• Built large-scale SaaS on top of
.NET
• Creator and maintainer of
Akka.NET since 2013
• Canonical actor model
implementation in .NET
• Highly concurrent, low-latency, and
distributed
• Used to build mission-critical real
time applications
• Performance is a feature

Garbage Collection
Perf Loss, GC Modes, and GC Generations

.NET GC
Source: https://www.csharpstar.com/interview-questions-garbage-collection-csharp/
Compacted
memory

GC Generations
The higher the generation, the most expensive the GC:
• Memory is more fragmented (access is random, not contiguous)
• Compaction takes longer (bigger gaps, more stuff to move,
longer GC pauses)

.NET Memory Model
private readonly Random myRandom = Random.Shared;
private void DoThing()
{
var i = myRandom.Next();
var j = myRandom.Next(i);
var b = i + j;
var str = b.ToString();
Console.WriteLine(str);
}
Stack
0xAEDC DoThing_vtable
0xFFBD ref(Random.Shared)
0x11CD i = 10;
0x11CE j = 5;
0x11CF b = 15;
0xADDE ref(string)
Managed Heap
0xAEDC class Thing_DoThing mthd
0xFFBD Random.Shared [1024b]
…
…
…
0xADDE string “15”

GC Considerations
• If you can: keep allocations in Gen 0 / 1
• Value types (no GC)
• Less memory fragmentation, compaction
• Less impact on latency, throughput
• If you can’t: keep Gen 2 objects in Gen 2 forever
• No GC if they’re still rooted!

GC Practice: Object Pools
• Microsoft.Extensions.ObjectPool<T> - great option for
long-lived Gen2 objects
• Best candidates are “reusable” types
• StringBuilder
• byte[](there are separate MemoryPool types for this)
• Use pre-allocated object, return to pool upon completion
• Doesn’t cause allocations so long as pool capacity isn’t exceeded

StringBuilder sb = null;
try
{
sb = _sbPool.Get();
using (var tw = new StringWriter(sb, CultureInfo.InvariantCulture))
{
var ser = JsonSerializer.CreateDefault(Settings);
ser.Formatting = Formatting.None;
using (var jw = new JsonTextWriter(tw))
{
ser.Serialize(jw, obj);
}
return Encoding.UTF8.GetBytes(tw.ToString());
}
}
finally
{
if (sb != null)
{
_sbPool.Return(sb);
}
}
Rent an instance from the
ObjectPool<StringBuilder>
Do our work
Return to the pool

• Pooling StringBuilder inside Newtonsoft.Json
~30% memory savings,
eliminated 100% of Gen 1
GC
~28% throughput
improvement in concurrent
use cases

Workstation GC vs. Server GC
<Project Sdk="Microsoft.NET.Sdk">
<Import Project="....common.props" />
<PropertyGroup>
<Description>Akka.Remote x-plat performance benchmark</Description>
<Copyright>Copyright (c) Akka.NET Team</Copyright>
<AssemblyTitle>RemotePingPong</AssemblyTitle>
<AssemblyName>RemotePingPong</AssemblyName>
<Authors>Akka.NET Team</Authors>
<TargetFrameworks>$(NetFrameworkTestVersion);$(NetTestVersion);$(NetCoreTestVersion)</TargetFrameworks>
<OutputType>Exe</OutputType>
</PropertyGroup>
<PropertyGroup>
<PlatformTarget>x64</PlatformTarget>
</PropertyGroup>
<PropertyGroup>
<ServerGarbageCollection>true</ServerGarbageCollection>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="....coreAkka.RemoteAkka.Remote.csproj" />
</ItemGroup>
</Project>
Not enabled by
default!

Workstation GC vs. Server GC
Workstation GC Server GC

Memory Allocations
Hidden Sources of Allocations and Eliminating Them

Allocations: Delegates and Closures
/// <summary>
/// Processes the contents of the mailbox
/// </summary>
public void Run()
{
try
{
if (!IsClosed()) // Volatile read, needed here
{
Actor.UseThreadContext(() =>
{
ProcessAllSystemMessages(); // First, deal with any system messages
ProcessMailbox(); // Then deal with messages
});
}
}
finally
{
SetAsIdle(); // Volatile write, needed here
Dispatcher.RegisterForExecution(this, false, false); // schedule to run again if there are more messages,
possibly
}
}
Critical path of actor msg
processing
Closes over ‘this’, allocates
delegate each time

Eliminate Delegate: Inlining
/// <summary>
/// Processes the contents of the mailbox
/// </summary>
public void Run()
{
try
{
if (!IsClosed()) // Volatile read, needed here
{
var tmp = InternalCurrentActorCellKeeper.Current;
InternalCurrentActorCellKeeper.Current = Actor;
try
{
ProcessAllSystemMessages(); // First, deal with any system messages
ProcessMailbox(); // Then deal with messages
}
finally
{
//ensure we set back the old context
InternalCurrentActorCellKeeper.Current = tmp;
}
}
}
finally
{
SetAsIdle(); // Volatile write, needed here
Dispatcher.RegisterForExecution(this, false, false); // schedule to run again if there are more messages, possibly
}
}
Eliminate delegate by inlining function
From 21kb & 203kb to ~1kb
Throughput improvement of ~10%

Other Delegate Allocation Removal Methods
• C#9: declare `static` delegates
• Cache delegates / use expression compiler
• ValueDelegates

Value Delegates
private readonly struct RequestWorkerTask : IRunnable
{
private readonly DedicatedThreadPoolTaskScheduler _scheduler;
public RequestWorkerTask(DedicatedThreadPoolTaskScheduler
scheduler)
{
_scheduler = scheduler;
}
public void Run()
{
// do work
}
}
private void RequestWorker()
{
_pool.QueueUserWorkItem(new RequestWorkerTask(this));
}
Implement our “delegate interface”
using a value type
Runs just the same as a reference
type
Execute the work (might cause a
boxing allocation!)

Allocations: “Empty” Collections
public State(TS stateName, TD stateData, TimeSpan? timeout = null, Reason stopReason = null, IReadOnlyList<object> replies = null, bool
notifies = true)
{
Replies = replies ?? new List<object>();
StopReason = stopReason;
Timeout = timeout;
StateData = stateData;
StateName = stateName;
Notifies = notifies;
}
Allocates a new,
non-empty array
(32 bytes)
Suspicious…

Allocations: “Empty” Collections
public State(TS stateName, TD stateData, TimeSpan? timeout = null, Reason stopReason = null, IReadOnlyList<object> replies = null, bool
notifies = true)
{
Replies = replies ?? Array.Empty<object>();
StopReason = stopReason;
Timeout = timeout;
StateName = stateName;
Notifies = notifies;
}
Creates an empty, non-
null collection

Value Types
Not Always Preferable Over Reference Types

Reference Type: FSM Events
public sealed class Event<TD> : INoSerializationVerificationNeeded
{
public Event(object fsmEvent, TD stateData)
{
FsmEvent = fsmEvent;
}
public object FsmEvent { get; }
public TD StateData { get; }
public override string ToString()
{
return $"Event: <{FsmEvent}>, StateData: <{StateData}>";
}
}
We allocate millions of these per
second in busy networks

public readonly struct Event<TD> : INoSerializationVerificationNeeded
{
public Event(object fsmEvent, TD stateData)
{
FsmEvent = fsmEvent;
}
public object FsmEvent { get; }
public TD StateData { get; }
public override string ToString()
{
return $"Event: <{FsmEvent}>, StateData: <{StateData}>";
}
}
Value Type: FSM Events
Change to value type
Reduction of ~30mb
Minor throughput
improvement

Value Types: Boxing Allocations
• Boxing occurs implicitly – when a
struct is cast into an object
• The struct will be wrapped
into an object and placed on
the managed heap.
• Unboxing happens explicitly –
when the object is cast back
into its associated value type.
• Can create a lot of allocations!
StateName is usually an enum (value
type) – is the object.Equals call
boxing?

Value Types: Boxing Allocations
// avoid boxing
if (!EqualityComparer<TState>.Default.Equals(_currentState.StateName, nextState.StateName) || nextState.Notifies)
{
_nextState = nextState;
HandleTransition(_currentState.StateName, nextState.StateName);
Listeners.Gossip(new Transition<TState>(Self, _currentState.StateName, nextState.StateName));
_nextState = default;
}
Used generic comparer to avoid casting
value types into object – removed 100%
of boxing allocations at this callsite.

Value Type: Message Envelope
/// <summary>
/// Envelope class, represents a message and the sender of the message.
/// </summary>
public readonly struct Envelope
{
public Envelope(object message, IActorRef sender)
{
Message = message;
Sender = sender;
}
public IActorRef Sender { get; }
public object Message { get; }
}
Used millions of times per
second in Akka.NET
readonly struct? Value
type? Should be “zero
allocations”

Reference Type: Message Envelope
/// <summary>
/// Envelope class, represents a message and the sender of the message.
/// </summary>
public sealed class Envelope
{
public Envelope(object message, IActorRef sender)
{
Message = message;
Sender = sender;
}
public IActorRef Sender { get; }
public object Message { get; }
}
What if we change to a
reference type? Will this reduce
allocations?
394kb  264kb
3.15mb  2.1mb
215 us  147 us
1860 us  1332 us

Value Type Pitfalls
• Copy-by-Value
• References to value types in other scopes requires copying
• ref parameters can work, but in narrowly defined contexts
• Excessive copying can be more expensive than allocating a reference
• Use reference types when semantics are “referential”
• Value types are not magic – work best in “tight” scopes
• Use the right tool for the job

NET Systems Programming Learned the Hard Way.pptx

Reference Type: Message Envelope
• What happens when we benchmark with significantly increased
cross-thread message traffic?
• Now if we convert Envelope back into a struct again…
• Thread access makes a difference!

Threads, Memory,
Synchronization, and Pain
Threads Hate You and Your Code

ThreadStatic and ThreadLocal<T>
• Allocates objects directly into thread local storage
• Objects stay there and are available each time thread is used
• Ideal for caching and pooling
• No synchronization
• Data and work all performed adjacent to stack memory
• Downside: thread-local data structures aren’t synchronized
• Variants!

Thread Local Storage & Context Switching
• Reference types passed between
threads often age into older
generations of GC
• Value types passed between
threads are copied (no GC)
• Thread-local state is copied into
CPU’s L1/L2 cache from memory
typically during execution
• Context switching occurs when
threads get scheduled onto
different CPUs or work is moves
onto different threads.

Thread Locality & Context Switching
Each thread gets
~30ms of execution
time before yielding

Current quantum is
over – time for
other threads to
have a turn

Context switch! Thread
0 now executing on CPU
1 – memory and state
will have to be
transferred.

Context Switching: High Latency Impact
/// <summary>
/// An asynchronous operation will be executed by
a <see cref="MessageDispatcher"/>.
/// </summary>
#if NETSTANDARD
public interface IRunnable
#else
public interface IRunnable : IThreadPoolWorkItem
#endif
{
/// <summary>
/// Executes the task.
/// </summary>
void Run();
}
// use native .NET 6 APIs here to reduce
allocations
// preferLocal to help reduce context switching
ThreadPool.UnsafeQueueUserWorkItem(run,
true);
IThreadPoolWorkItem interface
added in .NET Core 3.0 – avoids delegate
allocations for executing on ThreadPool
Consume IThreadPoolWorkItem
with preferLocal=true – tells the
ThreadPool to attempt to reschedule
work on current thread / CPU.

Performance Comparison
Before After
~3x improvement
~50% improvement

Thread Locality w/o Context Switching
No context switch –
same thread will have a
chance to execute on
same CPU. Might be
able to benefit from
L1/L2 cache, locality of
memory access, etc.

Data Structures & Synchronization
/// <summary> An unbounded mailbox message queue. </summary>
public class UnboundedMessageQueue : IMessageQueue, IUnboundedMessageQueueSemantics
{
private readonly ConcurrentQueue<Envelope> _queue = new ConcurrentQueue<Envelope>();
/// <inheritdoc cref="IMessageQueue"/>
public bool HasMessages
{
get { return !_queue.IsEmpty; }
}
/// <inheritdoc cref="IMessageQueue"/>
public int Count
{
get { return _queue.Count; }
}
….
}
Could, in theory, improve
memory performance by
replacing with a LinkedList (no
array segment allocations from
resizing)

/// <summary> An unbounded mailbox message queue. </summary>
public class UnboundedMessageQueue : IMessageQueue, IUnboundedMessageQueueSemantics
{
private readonly object s_lock = new object();
private readonly LinkedList<Envelope> _linkedList = new LinkedList<Envelope>();
public bool HasMessages
{
get
{
return Count > 0;
}
}
public int Count
{
get
{
lock (s_lock)
{
return _linkedList.Count;
}
}
}
….
Not a thread-safe data
structure, has to be synced-
with lock
Should offer better memory
performance than
ConcurrentQueue<T>
Wooooooof 🤮

• What went wrong there?
• ConcurrentQueue<T> is lock-free
• Uses volatile and atomic compare-and-swap operations
• i.e. Interlocked.CompareExchange
• Significantly less expensive, even on a single thread, than lock
• LinkedList<T> may not be all that memory efficient
• Internal data structure allocations per-insert rather than array block
allocations
• Better off rolling your own, probably

Learn More
• https://getakka.net/ - Akka.NET website, Discord, and source
• https://aaronstannard.com/ - my blog

NET Systems Programming Learned the Hard Way.pptx

Recommended

More Related Content

What's hot (20)

Similar to NET Systems Programming Learned the Hard Way.pptx (20)

More from petabridge (20)

Recently uploaded (20)

NET Systems Programming Learned the Hard Way.pptx

Editor's Notes