EasyNetQ: A Layered API

I had a great discussion today on the EasyNetQ mailing list about a pull request. It forced me to articulate how I view the EasyNetQ API as being made up of distinct layers, each with a different purpose.

EasyNetQ is a collection of components that provide services on top of the RabbitMQ.Client library. These do things like serialization, error handling, thread marshalling, connection management, etc. They are composed by a mini-IoC container. You can replace any component with your own implementation quite easily. So if you’d like XML serialization instead of the built in JSON, just write an implementation of ISerializer and register it with the container.

These components are fronted by the IAdvancedBus API. This looks a lot like the AMQP specification, and indeed you can run most AMQP methods from this API. The only AMQP concept that this API hides from you is channels. This is because channels are a confusing low-level concept that should never have been part of the AMQP specification in the first place. ‘Advanced’ is not a very good name for this API to be honest, ‘Iamqp’ would be much better.

Layered on top of the advanced API are a set of messaging patterns: Publish/Subscribe, Request/Response, and Send/Receive. This is the ‘opinionated’ part of EasyNetQ. It is our take on how such patterns should be implemented. There is very little flexibility; either you accept our way of doing things, or you don’t use it. The intention is that you, the user, don’t have to expend mental bandwidth re-inventing the same patterns; you don’t have to make choices every time you simply want to publish a message and subscribe to it. It’s designed to achieve EasyNetQ’s core goal of making working with RabbitMQ as easy as possible.

The patterns sit behind the IBus API. Once again, this is a poor name, it’s got very little to do with the concept of a message bus. A better name would be IPackagedMessagePatterns.

IBus is intended to work for 80% of users, 80% of the time. It’s not exhaustive. If the pattern you want to implement is not provided by IBus, then you should use IAdvancedBus. There’s no problem with doing this, and it’s how EasyNetQ is designed to be used.

I hope this explains the design philosophy behind EasyNetQ and why I push back against pull requests that add complexity to the IBus API. I see the ease-of-use aspect of EasyNetQ as its most important attribute. RabbitMQ is a superb piece of infrastructure and I want as many people in the .NET community to use it as possible.

EasyNetQ: Publishing Non-Persistent Messages

In AMQP, buried in the basic.properties object that gets sent along with each published message, there is a delivery_mode setting. You can set it to either ‘persistent’ (1), or ‘non-persistent’ (2). It controls whether a message is persisted to disk or not. In the AMQP spec:

“The server SHOULD respect the persistent property of basic messages and SHOULD make a best-effort to hold persistent basic messages on a reliable storage mechanism.”

Of course it’s pointless setting delivery_mode to ‘persistent’ if you’re not publishing to a durable queue.

By default EasyNetQ sets delivery_mode to persistent (1) when calling IBus.Publish. We make the assumption that people would want this safe behaviour out-of-the-box. However, it does introduce a performance hit, so if you don’t care about losing messages in the case of a server restart you should be able to change this behaviour.

From version 0.26.3, EasyNetQ has a new boolean connection string parameter ‘persistentMessages’. By default it is set to true, but if you don’t need persistent messages, but do need high performance, set it to false:

vas bus = RabbitHutch.CreateBus("host=localhost;persistentMessages=false");

This setting has no effect on the advanced API (IAdvancedBus) where you have access to basic.properties and are free to set delivery_mode on a message by message basis.

EasyNetQ: Replace the Internal DI Container

EasyNetQ, is made up of a collection of independent components. Internally it uses a tiny internal DI (IoC) container called DefaultServiceProvider. If you look at the code for the static RabbitHutch class that you use to create instances of the core IBus interface, you will see that it simply creates a new DefaultServiceProvider, registers all of EasyNetQ’s components, and then calls container.Resolve<IBus>() creating a new instance of IBus with its tree of dependencies supplied by the container:

public static IBus CreateBus(IConnectionConfiguration connectionConfiguration, Action<IServiceRegister> registerServices)
{
    Preconditions.CheckNotNull(connectionConfiguration, "connectionConfiguration");
    Preconditions.CheckNotNull(registerServices, "registerServices");

    var container = createContainerInternal();
    if (container == null)
    {
        throw new EasyNetQException("Could not create container. " + 
            "Have you called SetContainerFactory(...) with a function that returns null?");
    }

    registerServices(container);
    container.Register(_ => connectionConfiguration);
    ComponentRegistration.RegisterServices(container);

    return container.Resolve<IBus>();
}

But what if you want EasyNetQ to use your container of choice? From version 0.25 the RabbitHutch class provides a static method, SetContainerFactory, that allows you to register an alternative container factory method that provides whatever implementation of EasyNetQ.IContainer that you care to supply.

In this example we are using the Castle Windsor IoC container:

// register our alternative container factory
RabbitHutch.SetContainerFactory(() =>
    {
        // create an instance of Windsor
        var windsorContainer = new WindsorContainer();
        // wrap it in our implementation of EasyNetQ.IContainer
        return new WindsorContainerWrapper(windsorContainer);
    });
// now we can create an IBus instance, but it's resolved from
// windsor, rather than EasyNetQ's default service provider.
var bus = RabbitHutch.CreateBus("host=localhost");

Here is how I implemented WindsorContainerWrapper:

public class WindsorContainerWrapper : IContainer, IDisposable
{
    private readonly IWindsorContainer windsorContainer;

    public WindsorContainerWrapper(IWindsorContainer windsorContainer)
    {
        this.windsorContainer = windsorContainer;
    }

    public TService Resolve<TService>() where TService : class
    {
        return windsorContainer.Resolve<TService>();
    }

    public IServiceRegister Register<TService>(System.Func<IServiceProvider, TService> serviceCreator) 
        where TService : class
    {
        windsorContainer.Register(
            Component.For<TService>().UsingFactoryMethod(() => serviceCreator(this)).LifeStyle.Singleton
            );
        return this;
    }

    public IServiceRegister Register<TService, TImplementation>() 
        where TService : class 
        where TImplementation : class, TService
    {
        windsorContainer.Register(
            Component.For<TService>().ImplementedBy<TImplementation>().LifeStyle.Singleton
            );
        return this;
    }

    public void Dispose()
    {
        windsorContainer.Dispose();
    }
}

Note that all EasyNetQ services should be registered as singletons.

It’s important that you dispose of Windsor correctly. EasyNetQ doesn’t provide a Dispose method on IContainer, but you can access the container via the advanced bus (yes, this is new too), and dispose of windsor that way:

((WindsorContainerWrapper)bus.Advanced.Container).Dispose();
bus.Dispose();

Happy containerisation!

EasyNetQ’s Minimalist DI Container

I’ve been a long time fan of IoC (or DI) containers ever since I first discovered Castle Windsor back in 2007. I’ve used Windsor in every major project I’ve been involved in since then, and if you’d gone to a developer event in the late naughties, you may well have encountered me speaking about Windsor. Indeed, Seb Lambla had the cheek to call me ‘Windsor man’. I shall label him ‘Rest-a-man’ in revenge.

When I started working on EasyNetQ in 2011, I initially thought it would be a very simple lightweight wrapper around the RabbitMQ.Client library. The initial versions were very procedural, ‘just get it done’, script-ish code burps. But as it turned into a more serious library, I started to factor the different pieces into more SRPish classes using dependency injection. At this point I was doing poor-man’s dependency injection, with an initial piece of wire-up code in the RabbitHutch class.

As EasyNetQ started to gain some traction outside 15below, I began to get questions like, “how do I use a different serializer?” And “I don’t like your error handling strategy, how can I implement my own?” I was also starting to get quite bored of maintaining the ever-growing wire up code. The obvious solution was to introduce a DI container, but I was very reluctant to take a dependency on something like Windsor. Writing a library is a very different proposition than writing an application. Every dependency you introduce is a dependency that your user also has to take. Imagine you are a happily using AutoFac and suddenly Castle.Windsor appears in your code base, or even worse, you are an up-to-date Windsor user, but EasyNetQ insists on installing an old version of Windsor alongside. Nasty. Windsor is an amazing library, some of its capabilities are quite magical, but I didn’t need any of these advanced features in EasyNetQ. In fact I could be highly constrained in my DI container requirements:

• I only needed singleton instances.
• The lifetime of all DI provided components matches the lifetime of the main IBus interface.
• I didn’t need the container to manage component disposal.
• I didn’t need open generic type registration.
• I was happy to explicitly register all my components, so I didn’t need convention based registration.
• I only needed constructor injection.
• I can guarantee that a component implementation will only have a single constructor.

With this highly simplified list of requirements, I realised that I could write a very simple DI container in just few lines of code (currently 113 as it turns out).

My super simple container only provides two registration methods. The first takes a service interface type and a instance creation function:

IServiceRegister Register<TService>(Func<IServiceProvider, TService> serviceCreator) where TService : class;

The second takes an instance type and an implementation type:

IServiceRegister Register<TService, TImplementation>()
    where TService : class
    where TImplementation : class, TService;

These are both defined in a IServiceRegister interface. There is also a single Resolve method in an IServiceProvider interface:

TService Resolve<TService>() where TService : class;

You can see all 113 lines of the implementation in the DefaultServiceProvider class. As you can see, it’s not at all complicated, just three dictionaries, one holding the list of service factories, another holding a list of registrations, and the last a list of instances. Each registration simply adds a record to the instances dictionary. When Resolve is called, a bit of reflection looks up the implementation type’s constructor and invokes it, recursively calling resolve for each constructor argument. If the service is provided by a service factory, it is invoked instead of the constructor.

I didn’t have any worries about performance. The registration and resolve code is only called once when a new instance of IBus is created. EasyNetQ is designed to be instantiated at application start-up and for that instance to last the lifetime of the application. For 90% of applications, you should only need a single IBus instance.

EasyNetQ’s component are registered in a ComponentRegistration class. This provides an opportunity for the user to register services ahead of the default registration, and since the first to register wins, it provides an easy path for users to replace default services implementations with their own. Here’s an example of replacing EasyNetQ’s default console logger with a custom logger:

var logger = new MyLogger();
var bus = RabbitHutch.CreateBus(connectionString, x => x.Register<IEasyNetQLogger>(_ => logger));

You can replace pretty much any of the internals of EasyNetQ in this way: the logger, the serializer, the error handling strategy, the dispatcher threading model … Check out the ComponentRegistration class to see the whole list.

Of course the magic of DI containers, the first rule of their use, and the thing that some people find extraordinarily hard to grok, is that I only need one call to Resolve in the entire EasyNetQ code base at line 146 in the RabbitHutch class:

return serviceProvider.Resolve<IBus>();

IoC/DI containers are often seen as very heavyweight beasts that only enterprise architecture astronauts could love, but nothing could be more wrong. A simple implementation only needs a few lines of code, but provides your application, or library, with considerable flexibility.

EasyNetQ is open source under the (very flexible) MIT licence. Feel free to cut-n-paste my DefaultServiceProvider class for your own use.

EasyNetQ: Non-Generic Subscribe

Since it’s very first version, EasyNetQ has allowed you to subscribe to a message simply by providing a handler for a given message type (and a subscription id, but that’s another discussion).

bus.Subscribe<MyMessage>("subscriptionId", x => Console.WriteLine(x.Text));

But what do you do if you are discovering the message type at runtime? For example, you might have some system which loads add-ins and wants to subscribe to message types on their behalf. Before today (version 0.24) you would have had to employ some nasty reflection mojo to deal with this scenario, but now EasyNetQ provides you with non-generic subscription methods out-of-the-box.

Just add the this using statement:

using EasyNetQ.NonGeneric;

Which provides you with these extension methods:

public static IDisposable Subscribe(this IBus bus, Type messageType, string subscriptionId, Action<object> onMessage)
public static IDisposable Subscribe(
    this IBus bus, 
    Type messageType, 
    string subscriptionId, 
    Action<object> onMessage, 
    Action<ISubscriptionConfiguration> configure)
public static IDisposable SubscribeAsync(
    this IBus bus,
    Type messageType,
    string subscriptionId,
    Func<object, Task> onMessage)
public static IDisposable SubscribeAsync(
    this IBus bus, 
    Type messageType, 
    string subscriptionId, 
    Func<object, Task> onMessage, 
    Action<ISubscriptionConfiguration> configure)

They are just like the Subscribe methods on IBus except that you provide a Type argument instead of the generic argument, and the message handler is an Action<object> instead of an Action<T>.

Here’s an example of the non-generic subscribe in use:

var messageType = typeof(MyMessage);
bus.Subscribe(messageType, "my_subscriptionId", x =>
    {
        var message = (MyMessage)x;
        Console.Out.WriteLine("Got Message: {0}", x.Text);
    });

Very useful I think, and one of the most commonly asked for features.

 

Happy runtime discovery!

EasyNetQ: Send Receive Pattern

From version 0.22, EasyNetQ supports a new message pattern: Send/Receive.

Whereas the Publish/Subscribe and Request/Response patterns are location transparent, in that you don't need to specify where the consumer of the message is located, the Send/Receive pattern is specifically designed for communication via a named queue. It also makes no assumptions about the types of message that can be sent via the queue. This means that you can send different types of message via the same queue.

The send/receive pattern is ideal for creating 'command pipelines', where you want a buffered channel to a single command processor.

To send a message, use the Send method on IBus, specifying the name of the queue you wish to sent the message to and the message itself:

bus.Send("my.queue", new MyMessage{ Text = "Hello Widgets!" });

To setup a message receiver for a particular message type, use the Receive method on IBus:

bus.Receive<MyMessage>("the.queue", message => Console.WriteLine("MyMessage: {0}", message.Text));

You can set up multiple receivers for different message types on the same queue by using the Receive overload that takes an Action<IReceiveRegistration>. For example, this sets up a receive for both MyMessage and MyOtherMessage:

bus.Receive("the.queue", x => x
    .Add<MyMessage>(message => deliveredMyMessage = message)
    .Add<MyOtherMessage>(message => deliveredMyOtherMessage = message));

If a message arrives on a receive queue that doesn't have a matching receiver, EasyNetQ will write the message to the EasyNetQ error queue with an exception saying 'No handler found for message type <the message type>'.

Note: You probably do not want to call bus.Receive more than once for the same queue. This will create a new consumer on the queue and RabbitMQ will round-robin between them. If you are consuming different types on different Receive calls (and thus different consumers), some of your messages will end up on the error queue because EasyNetQ will not find a handler for your message type associated with the consumer on which it is consumed.

EasyNetQ: Multiple Handlers per Consumer

A common feature request for EasyNetQ has been to have some way of implementing a command pipeline pattern. Say you’ve got a component that is emitting commands. In a .NET application each command would most probably be implemented as a separate class. A command might look something like this:

public class AddUser
{
    public string Username { get; private set; }
    public string Email { get; private set; }

    public AddUser(string username, string email) 
    {
        Username = username;
        Email = email;
    }
}

Another component might listen for commands and act on them. Previously in EasyNetQ it would have been difficult to implement this pattern because a consumer (Subscriber) was always bound to a given message type. You would have had to use the lower level IAdvancedBus binary message methods and implement your own serialization and dispatch infrastructure.

But now EasyNetQ comes with multiple handlers per consumer out of the box.

From version 0.20 there’s a new overload of the Consume method that provides a fluent way for you to register multiple message type handlers to a single consumer, and thus a single queue.

Here’s an example:

bus = RabbitHutch.CreateBus("host=localhost");

var queue = bus.Advanced.QueueDeclare("multiple_types");

bus.Advanced.Consume(queue, x => x
        .Add<AddUser>((message, info) => 
            { 
                Console.WriteLine("Add User {0}", message.Body.Username);
            })
        .Add<DeleteUser>((message, info) =>
            {
                Console.WriteLine("Delete User {0}", message.Body.Username);
            })
    );

Now we can publish multiple message types to the same queue:

bus.Advanced.Publish(Exchange.GetDefault(), queue.Name, false, false, 
    new Message<AddUser>(new AddUser("Steve Howe", "[email protected]"))));
bus.Advanced.Publish(Exchange.GetDefault(), queue.Name, false, false, 
    new Message<DeleteUser>(new DeleteUser("Steve Howe")));

By Default, if a matching handler cannot be found for a message, EasyNetQ will throw an exception. You can change this behaviour, and simply ignore messages that do not have a handler, by setting the ThrowOnNoMatchingHandler property to false, like this:

bus.Advanced.Consume(queue, x => x
        .Add<AddUser>((message, info) => 
            { 
                Console.WriteLine("Add User {0}", message.Body.Username);
            })
        .Add<DeleteUser>((message, info) =>
            {
                Console.WriteLine("Delete User {0}", message.Body.Username);
            })
        .ThrowOnNoMatchingHandler = false
    );

Very soon there will be a send/receive pattern implemented at the IBus level to make this even easier. Watch this space!

Happy commanding!

EasyNetQ: Consumer Cancellation

Consumer cancellation has been a requested feature of EasyNetQ for a while now. I wasn’t intending to implement it immediately, but a pull request by Daniel White today made me look at the whole issue of consumer cancellation from the point of view of a deleted queue. It led rapidly to a quite straightforward implementation of user cancellations. The wonders of open source software development, and the generosity of people like Daniel, never fails to impress me.

So what is consumer cancellation? It means that you can stop consuming from a queue without having dispose of the entire IBus instance and close the connection. All the IBus Subscribe methods, and the IAdvancedBus Consume methods now return an IDisposable. To stop consuming, just call Dispose like this:

var cancelSubscription = bus.Subscribe<MyMessage>("subscriptionId", MessageHandler);
.
.
// sometime later stop consuming
cancelSubscription.Dispose();

Nice :)

EasyNetQ: Changes to Conventions With Version 0.18

TL:DR: Exchange and queue names used to have ‘.’ replaced with ‘_’. From version 0.18 this is no longer the case.

Yesterday I announced version 0.18 of EasyNetQ. The big change was the addition of polymorphic publish and subscribe.

I forgot to mention that there’s a slight change to the conventions that EasyNetQ uses, that might affect you when you upgrade.

EasyNetQ’s publish-subscribe pattern is implemented in AMQP as follows:

• A topic exchange named after the published type is created.
• A queue named by concatenating the published type and the subscriber id is created.
• The exchange is bound to the queue with the wildcard, ‘#’, binding key.

So if you have this code:

bus.Subscribe<MyMessage>("test", MessageHandler);
bus.Publish<MyMessage>(message);

Note, in my case MyMessage is in an assembly ‘EasyNetQ.Tests’ with the same namespace.

You will see this in the RabbitMQ management UI:

img title="default_binding" style="border-left-width: 0px; border-right-width: 0px; border-bottom-width: 0px; display: inline; border-top-width: 0px" border="0" alt="default_binding" src="$default_binding5.png" width="715" height="487" />

Previously The exchange and queue names would have had the ‘.’ replaced with ‘_’: ‘EasyNetQ_Tests_MyMessage:EasyNetQ_tests’.

If you upgrade some services to version 0.18, but leave others at earlier version numbers, they won’t be able to communicate.

EasyNetQ: Polymorphic Publish and Subscribe

From version 18.0 of EasyNetQ, you can now subscribe to an interface, then publish implementations of that interface.

Let's look at an example. I have an interface IAnimal and two implementations Cat and Dog:

public interface IAnimal
{
    string Name { get; set; }
}

public class Cat : IAnimal
{
    public string Name { get; set; }
    public string Meow { get; set; }
}

public class Dog : IAnimal
{
    public string Name { get; set; }
    public string Bark { get; set; }
}

I can subscribe to IAnimal and receive both Cat and Dog classes, or any other class that implements IAnimal:

bus.Subscribe<IAnimal>("polymorphic_test", @interface =>
    {
        var cat = @interface as Cat;
        var dog = @interface as Dog;

        if (cat != null)
        {
            Console.Out.WriteLine("Name = {0}", cat.Name);
            Console.Out.WriteLine("Meow = {0}", cat.Meow);
        }
        else if (dog != null)
        {
            Console.Out.WriteLine("Name = {0}", dog.Name);
            Console.Out.WriteLine("Bark = {0}", dog.Bark);
        }
        else
        {
            Console.Out.WriteLine("message was not a dog or a cat");
        }
    });

Let's publish a cat and a dog:

var cat = new Cat
{
    Name = "Gobbolino",
    Meow = "Purr"
};

var dog = new Dog
{
    Name = "Rover",
    Bark = "Woof"
};

bus.Publish<IAnimal>(cat);
bus.Publish<IAnimal>(dog);

Note that I have to explicitly specify that I am publishing IAnimal. EasyNetQ uses the generic type specified in the Publish and Subscribe methods to route the publications to the subscriptions.

Happy polymorphismising!

EasyNetQ: Big Breaking Changes to Request-Response

My intensive work on EasyNetQ (our super simple .NET API for RabbitMQ) continues. I’ve been taking lessons learned from nearly two years of production and the fantastic feedback from EasyNetQ’s users, mashing this together, and making lots changes to both the internals and the API. I know that API changes cause problems for users; they break your application and force you to revisit your code. But the longer term benefits should outweigh the immediate costs as EasyNetQ slowly morphs into a solid, reliable library that really does make working with RabbitMQ as easy as possible.

Changes in version 0.17 are all around the request-response pattern. The initial implementation was very rough with lots of nasty ways that resource use could run away when things went wrong. The lack of timeouts also meant that your application could wait forever when messages got lost. Lastly the API was quite clunky, with call-backs where Tasks are a far better choice. All these problems have been corrected in this version.

API changes

There is now a synchronous Request method. Of course messaging is by nature a asynchronous operation, but sometimes you just want the simplest possible thing and you don’t care about blocking your thread while you wait for a response. Here’s what it looks like:

var response = bus.Request<TestRequestMessage, TestResponseMessage>(request);

The old call-back Request method has been removed. There was no need for it when the RequestAsync that returned a Task<TResult> was always a better choice:

var task = bus.RequestAsync<TestRequestMessage, TestResponseMessage>(request)

task.ContinueWith(response =>
{
    Console.WriteLine("Got response: '{0}'", response.Result.Text);
});

Timeouts

Timeouts are an essential ingredient of any distributed system. This probably deserves a blog post of its own, but no matter how resilient you make your architecture, if an important piece simply goes away (like the network for example), you need a circuit breaker. EasyNetQ now has a global timeout that you can configure via the connection string:

var bus = RabbitHutch.CreateBus("host=localhost;timeout=60");

Here we’ve configured the timeout as 60 seconds. The default is 10 seconds. If you make a request, but no response is received within the timeout period, a System.Timeout exception will be thrown.

If the connection goes away while a request is in-flight, EasyNetQ doesn’t wait for the timeout to fire, but immediately throws an EasyNetQException with a message saying that the connection has been lost. Your application should catch both Timeout and EasyNetQ exceptions and react appropriately.

Internal Changes

My last blog post was a discussion of the implementation options of request-response with RabbitMQ. As I said there, I now believe that a single exclusive queue for all responses to a client is the best option. Version 0.17 implements this. When you call bus.Request(…) you will see a queue created named easynetq.response.<some guid>. This will last for the lifetime of the current connection.

Happy requesting!

RabbitMQ Request-Response Pattern

If you are programming against a web service, the natural pattern is request-response. It’s always initiated by the client, which then waits for a response from the server. It’s great if the client wants to send some information to a server, or request some information based on some criteria. It’s not so useful if the server wants to initiate the send of some information to the client. There we have to rely on somewhat extended HTTP tricks like long-polling or web-hooks.

With messaging systems, the natural pattern is send-receive. A producer node publishes a message which is then passed to a consuming node. There is no real concept of client or server; a node can be a producer, a consumer, or both. This works very well when one node wants to send some information to another or vice-versa, but isn’t so useful if one node wants to request information from another based on some criteria.

All is not lost though. We can model request-response by having the client node create a reply queue for the response to a query message it sends to the server. The client can set the request message properties’ reply_to field with the reply queue name. The server inspects the reply_to field and publishes the reply to the reply queue via the default exchange, which is then consumed by the client.

The implementation is simple on the request side, it looks just like a standard send-receive. But on the reply side we have some choices to make. If you Google for ‘RabbitMQ RPC’, or ‘RabbitMQ request response’, you will find several different opinions concerning the nature of the reply queue.

• Should there be a reply queue per request, or should the client maintain a single reply queue for multiple requests?
• Should the reply queue be exclusive, only available to this channel, or not? Note that an exclusive queue will be deleted when the channel is closed, either intentionally, or if there is a network or broker failure that causes the connection to be lost.

Let’s have a look at the pros and cons of these choices.

Exclusive reply queue per request.

Here each request creates a reply queue. The benefits are that it is simple to implement. There is no problem with correlating the response with the request, since each request has its own response consumer. If the connection between the client and the broker fails before a response is received, the broker will dispose of any remaining reply queues and the response message will be lost.

The main implementation issue is that we need to clean up any replies queues in the event that a problem with the server means that it never publishes the response.

This pattern has a performance cost because a new queue and consumer has to be created for each request.

Exclusive reply queue per client

Here each client connection maintains a reply queue which many requests can share. This avoids the performance cost of creating a queue and consumer per request, but adds the overhead that the client needs to keep track of the reply queue and match up responses with their respective requests. The standard way of doing this is with a correlation id that is copied by the server from the request to the response.

Once again, there is no problem with deleting the reply queue when the client disconnects because the broker will do this automatically. It does mean that any responses that are in-flight at the time of a disconnection will be lost.

Durable reply queue

Both the options above have the problem that the response message can be lost if the connection between the client and broker goes down while the response is in flight. This is because they use exclusive queues that are deleted by the broker when the connection that owns them is closed.

The natural answer to this is to use a non-exclusive reply queue. However this creates some management overhead. You need some way to name the reply queue and associate it with a particular client. The problem is that it’s difficult for the client to know if any one reply queue belongs to itself, or to another instance. It’s easy to naively create a situation where responses are being delivered to the wrong instance of the client. You will probably wind up manually creating and naming response queues, which removes one of the main benefits of choosing broker based messaging in the first place.

 EasyNetQ

For a high-level re-useable library like EasyNetQ the durable reply queue option is out of the question. There is no sensible way of knowing whether a particular instance of the library belongs to a single logical instance of a client application. By ‘logical instance’ I mean an instance that might have been stopped and started, as opposed to two separate instances of the same client.

Instead we have to use exclusive queues and accept the occasional loss of response messages. It is essential to implement a timeout, so that an exception can be raised to the client application in the event of response loss. Ideally the client will catch the exception and re-try the message if appropriate.

Currently EasyNetQ implements the ‘reply queue per request’ pattern, but I’m planning to change it to a ‘reply queue per client’. The overhead of matching up responses to requests is not too onerous, and it is both more efficient and easier to manage.

I’d be very interested in hearing other people’s experiences in implementing request-response with RabbitMQ.

EasyNetQ: Publisher Confirms

Publisher confirms are a RabbitMQ addition to AMQP to guarantee message delivery. You can read all about them here and here. In short they provide a asynchronous confirmation that a publish has successfully reached all the queues that it was routed to.

To turn on publisher confirms with EasyNetQ set the publisherConfirms connection string parameter like this:

var bus = RabbitHutch.CreateBus("host=localhost;publisherConfirms=true");

When you set this flag, EasyNetQ will wait for the confirmation, or a timeout, before returning from the Publish method:

bus.Publish(new MyMessage
    {
        Text = "Hello World!"
    });
// here the publish has been confirmed.

Nice and easy.

There’s a problem though. If I run the above code in a while loop without publisher confirms, I can publish around 4000 messages per second, but with publisher confirms switched on that drops to around 140 per second. Not so good.

With EasyNetQ 0.15 we introduced a new PublishAsync method that returns a Task. The Task completes when the publish is confirmed:

bus.PublishAsync(message).ContinueWith(task =>
    {
        if (task.IsCompleted)
        {
            Console.WriteLine("Publish completed fine.");
        }
        if (task.IsFaulted)
        {
            Console.WriteLine(task.Exception);
        }
    });

Using this code in a while loop gets us back to 4000 messages per second with publisher confirms on.

Happy confirms!

EasyNetQ: Big Breaking Changes in the Publish API

From version 0.15 the way that publish works in EasyNetQ has dramatically changed. Previously the client application was responsible for creating and disposing the AMQP channel for the publication, something like this:

using (var channel = bus.OpenPublishChannel())
{
    channel.Publish(message);
}

There are several problems with this approach.

The practical ones are that it encourages developers to use far more channels than they need to. The codebases that I’ve looked at often have the pattern exactly as it’s given above, even if publish is invoked in a loop. Channel creation is relatively cheap, but it’s not free and frequent channel creation imposes a cost on the broker. However, if a developer tries to keep a channel open for a number of publish calls, they then have to deal with the complex scenario of recovering from a connection loss.

The more conceptual, design oriented problem, is that it fails in terms of EasyNetQ’s mission statement, which is to make building .NET applications with RabbitMQ as easy as possible. With the core (IBus) API, the developer shouldn’t have to be concerned about AMQP specifics like channel handling, the library should do all that for them.

From version 0.15, you don’t need to open a publish channel, simply call the new Publish method directly on the IBus interface:

bus.Publish(message);

Internally EasyNetQ maintains a single channel for all outgoing AMQP calls and marshals all client invocations onto a single internally maintained thread. So while EasyNetQ is thread-safe, all internal calls to the RabbitMQ.Client library are serialised. Consumers haven’t changed and are invoked from a separate thread.

The Request call has also been moved to the IBus API:

bus.Request<TestRequestMessage, TestResponseMessage>(new TestRequestMessage {Text = "Hello World!"},
    response => Console.WriteLine("Got response: " + response.Text));

The change also means that EasyNetQ can take full responsibility for channel reconnection in the event of connection failure and leads to a much nicer publisher confirms implementation which I’ll be blogging about soon.

Happy messaging!

EasyNetQ: IConsumerDispatcher

EasyNetQ has always had a single dispatcher thread that runs user message handlers. This means that a slow handler on one queue can cause other handlers on other queues to wait. The intention is that one shouldn’t write long running consumers. If you are doing long running IO you should use the SubsribeAsync method and return a task that completes when the long running IO completes.

A recent change to EasyNetQ has been to make the dispatcher a separate abstraction. This means you can replace it with your own implementation if desired.

Inside EasyNetQ the dispatcher receives deliveries from the RabbitMQ C# Client library and places the delivery information on an internal concurrent queue. By default, all consumers share a single internal queue. A single dispatcher thread pulls deliveries from the queue and then asks the consumer to invoke the user message handler.

The consumer dispatcher implementation is very simple, it simply maintains a queue of Action and a thread which takes those actions from the end of the queue and runs them. You could use the same pattern whenever you need to marshal an action onto a single thread. I wrote about this more general terms here.

public class ConsumerDispatcher : IConsumerDispatcher
{
    private readonly Thread dispatchThread;
    private readonly BlockingCollection<Action> queue = new BlockingCollection<Action>();
    private bool disposed;

    public ConsumerDispatcher(IEasyNetQLogger logger)
    {
        Preconditions.CheckNotNull(logger, "logger");

        dispatchThread = new Thread(_ =>
            {
                try
                {
                    while (true)
                    {
                        if (disposed) break;

                        queue.Take()();
                    }
                }
                catch (InvalidOperationException)
                {
                    // InvalidOperationException is thrown when Take is called after 
                    // queue.CompleteAdding(), this is signals that this class is being
                    // disposed, so we allow the thread to complete.
                }
                catch (Exception exception)
                {
                    logger.ErrorWrite(exception);
                }
            }) { Name = "EasyNetQ consumer dispatch thread" };
        dispatchThread.Start();
    }

    public void QueueAction(Action action)
    {
        Preconditions.CheckNotNull(action, "action");
        queue.Add(action);
    }

    public void Dispose()
    {
        queue.CompleteAdding();
        disposed = true;
    }
}

An implementation of IConsumerDispatcherFactory maintains a single instance of IConsumerDispatcher:

public class ConsumerDispatcherFactory : IConsumerDispatcherFactory
{
    private readonly Lazy<IConsumerDispatcher> dispatcher;

    public ConsumerDispatcherFactory(IEasyNetQLogger logger)
    {
        Preconditions.CheckNotNull(logger, "logger");
        
        dispatcher = new Lazy<IConsumerDispatcher>(() => new ConsumerDispatcher(logger));
    }

    public IConsumerDispatcher GetConsumerDispatcher()
    {
        return dispatcher.Value;
    }
    
    public void Dispose()
    {
        if (dispatcher.IsValueCreated)
        {
            dispatcher.Value.Dispose();
        }
    }
}

If you wanted to use an alternative dispatch strategy; say for example you (quite reasonably) wanted a new dispatch thread for each consumer, you would simply implement an alternative IConsumerDispatcherFactory and register it with EasyNetQ like this:

var bus = RabbitHutch.CreateBus("host=localhost", 
    x => x.Register<IConsumerDispatcherFactory>(_ => new MyConsumerDispatcherFactory()));

Happy messaging!

RabbitMQ: AMQP Channel Best Practices

I’ve long been confused with best practices around AMQP channel handling. This post is my attempt to explain my current thoughts, partly in an attempt to elicit feedback.

The conversation between a message broker and a client is two-way. Both the client and the server can initiate ‘communication events’; the client can invoke a method on the server: ‘publish’ or ‘declare’, for example; and the server can invoke a method on the client such as ‘deliver’ or ‘reject’. Because the client needs to receive methods from the server, it needs to keep a connection open for its lifetime. This means that broker connections may last long periods; hours, days, or weeks. Maintaining these connections is expensive, both for the client and the server. In order to have many logical connections without the overhead of many physical TCP/IP connections, AMQP has the concept of ‘channel’ (confusingly represented by the IModel interface in the Java and .NET clients). You can create multiple channels on a single connection and it’s relatively cheap to create and dispose of them.

But what are the recommended rules for handling these channels? Should I create a new one for every operation? Or should I just keep a single one around and execute every method on it?

First some hard and fast rules (note I’m using the RabbitMQ .NET client for reference here, other clients might have different behaviour):

Channels are not thread safe. You should create and use a channel on a single thread. From the .NET client documentation (section 2.10):

“In general, IModel instances should not be used by more than one thread simultaneously: application code should maintain a clear notion of thread ownership for IModel instances.”

Apparently this is not a hard and fast rule with the Java client, just good advice. The java client serializes all calls to a channel.

An ACK should be invoked on the same channel on which the delivery was received. The delivery tag is scoped to the channel and an ACK sent to a different channel from which the delivery was received will cause a channel error. This somewhat contradicts the ‘Channels are not thread safe’ directive above since you will probably ACK on a different thread from one where you invoked basic.consume, but this seems to be acceptable.

Now some softer suggestions:

It seems neater to create a channel for each consumer. I like thinking of the consumer and channel as a unit: when I create a consumer, I also create a channel for it to consume from. When the consumer goes away, so does the channel and vice-versa.

Do not mix publishing and consuming on the same channel. If you follow the suggestion above, it implies that you have dedicated channels for each consumer. It follows that you should create separate channels for server and client originated methods.

Maintain a long running publish channel. My current implementation of EasyNetQ makes creating channels for publishing the responsibility of the user. I now think this is mistake. It encourages the pattern of: create a channel, publish, dispose the channel. This pattern doesn’t work with publisher confirms where you need to keep the channel open at least until you receive an ACK or NACK. And although creating channels is relatively lightweight, there is still some overhead. I now favour the approach of maintaining a single publish channel on a dedicated thread and marshalling publish (and declare) calls to it. This is potentially a bottleneck, but the impressive performance of non-transactional publishing means that I’m not overly concerned about it.

EasyNetQ: Topic Confusion!

This is a quick post to highlight a common cause of confusion when people play with topics in EasyNetQ.

You can subscribe to a message type with a topic like this:

bus.Subscribe<MyMessage>("id1", myHandler, x => x.WithTopic("X.*"));

Topics are dot separated strings that match with the routing key attached to a message at publication. In the code above I’ve said, give me any message of type MyMessage who’s topic matches “x.*”. The ‘*’ character is a wildcard, so “X.A” would match, as would “X.Z”, but “Y.A” wouldn’t.

You publish with a topic like this:

using (var publishChannel = bus.OpenPublishChannel())
{
    publishChannel.Publish(myMessage, x => x.WithTopic("X.A"));
}

The confusion occurs when the topic in the subscribe call changes. Maybe you are experimenting with topics by changing the string in the WithTopic( … ) method, or perhaps you are hoping to dynamically change the topic at runtime? Maybe you’ve done several subscribes, each with a different handler and a different topic, but the same subscription Id. Either way, you’ll probably be surprised to find that you still get all the messages matched by previously set topics as well as those matched by the current topic.

In order to explain why this happens, let’s look at how EasyNetQ creates exchanges, queues and bindings when you call these API methods.

If I call the subscribe method above, EasyNetQ will declare a topic-exchange named after the type, a queue named after the type and the subscription id, and a bind them with the given topic:

If I change the topic and run the code again like this:

bus.Subscribe<MyMessage>("id1", myHandler, x => x.WithTopic("Y.*"));

EasyNetQ will declare the same queue and exchange. The word ‘declare’ is the key here, it means, “if the object doesn’t exist, create it, otherwise do nothing.” The queue and exchange already exist, so declaring them has no effect. EasyNetQ then binds them with the given routing key. The original binding still exists – we haven’t done anything to remove it – so we now have two bindings with two different topics between the exchange and the queue:

This means that any message matching ‘X.*’ or ‘Y.*’ will be routed to the queue, and thus to our consumer.

So, beware when playing with topics!

As an aside, the ability to create multiple bindings is a very powerful feature. It allows you to implement ‘OR’ semantics for routing messages. If this is what you want, you should concatenate multiple WithTopic methods rather than make multiple calls to Subscribe. For example, say we wanted to implement ‘*.B OR Y.*’:

bus.Subscribe<MyMessage>("id4", myHandler, x => x.WithTopic("Y.*").WithTopic("*.B"));

Which would give us the desired result:

Happy routing!

EasyNetQ: Big Breaking Changes in the Advanced Bus

EasyNetQ is my little, easy to use, client API for RabbitMQ. It’s been doing really well recently. As I write this it has 24,653 downloads on NuGet making it by far the most popular high-level RabbitMQ API.
The goal of EasyNetQ is to make working with RabbitMQ as easy as possible. I wanted junior developers to be able to use basic messaging patterns out-of-the-box with just a few lines of code and have EasyNetQ do all the heavy lifting: exchange-binding-queue configuration, error management, connection management, serialization, thread handling; all the things that make working against the low level AMQP C# API, provided by RabbitMQ, such a steep learning curve.
To meet this goal, EasyNetQ has to be a very opinionated library. It has a set way of configuring exchanges, bindings and queues based the .NET type of your messages. However, right from the first release, many users said that they liked the connection management, thread handling, and error management, but wanted to be able to set up their own broker topology. To support this we introduced the advanced API, an idea stolen shamelessly from Ayende’s RavenDb client.
You access the advanced bus (IAdvancedBus) via the Advanced property on IBus:

var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced;

Sometimes something can seem like a good idea at the time, and then later you think, “WTF! Why on earth did I do that?” It happens to me all the time. I thought it would be cool if one created the exchange-binding-queue topology and then passed it to the publish and subscribe methods, which would then internally declare the exchanges and queues and do the binding. I implemented a tasty little visitor pattern in my ITopologyVisitor. I optimised for (my) programming fun rather than an a simple, obvious, easy to understand API.
I realised a while ago that a more straightforward set of declares on IAdvancedBus would be a far more obvious and intentional design. To this end, I’ve refactored the advanced bus to separate declares from publishing and consuming. I just pushed the changes to NuGet and have also updated the Advanced Bus documentation. Note these are breaking changes, so please be careful if you are upgrading to the latest version, 0.12, and upwards.
Here are some tasters of how it works:
Declare a queue, exchange and binding, and consume raw message bytes:

var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced;

var queue = advancedBus.QueueDeclare("my_queue");
var exchange = advancedBus.ExchangeDeclare("my_exchange", ExchangeType.Direct);
advancedBus.Bind(exchange, queue, "routing_key");

advancedBus.Consume(queue, (body, properties, info) => Task.Factory.StartNew(() =>
    {
        var message = Encoding.UTF8.GetString(body);
        Console.Out.WriteLine("Got message: '{0}'", message);
    }));

Note I’ve renamed ‘Subscribe’ to ‘Consume’ to better reflect the underlying AMQP method.
Declare an exchange and publish a message:

var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced;

var exchange = advancedBus.ExchangeDeclare("my_exchange", ExchangeType.Direct);

using (var channel = advancedBus.OpenPublishChannel())
{
    var body = Encoding.UTF8.GetBytes("Hello World!");
    channel.Publish(exchange, "routing_key", new MessageProperties(), body);
}

You can also delete exchanges, queues and bindings:

var advancedBus = RabbitHutch.CreateBus("host=localhost").Advanced;

// declare some objects
var queue = advancedBus.QueueDeclare("my_queue");
var exchange = advancedBus.ExchangeDeclare("my_exchange", ExchangeType.Direct);
var binding = advancedBus.Bind(exchange, queue, "routing_key");

// and then delete them
advancedBus.BindingDelete(binding);
advancedBus.ExchangeDelete(exchange);
advancedBus.QueueDelete(queue);

advancedBus.Dispose();

I think these changes make for a much better advanced API. Have a look at the documentation for the details.

EasyNetQ on .NET Rocks!

Last week I had the pleasure of being interviewed by Carl Franklin and Richard Campbell for a .NET Rocks episode on RabbitMQ and EasyNetQ. It was terrific fun and a real honour to be invited on the show. I’ve been listening to .NET Rocks since it started in 2002 so you can imagine how excited I was. Carl and Richard are seasoned pros when it comes to podcasting, and the awesome ninja editing skills they posses turned my rather hesitant and rambling answers into something that almost sounded coherent.
You can listen to the show on the link below:
http://www.dotnetrocks.com/default.aspx?ShowNum=848
Now Richard, about that Tesla …

EasyNetQ in Africa

Anthony Moloney got in touch with me recently. He’s using EasyNetQ, my simple .NET API for RabbitMQ, with his team in Kenya. Here’s an email he sent me:

Hi Mike,
Further to the brief twitter exchange today about using EasyNetQ on our Kenyan project. We started using EasyNetQ back in early November and I kept meaning to drop you a line to thank you for all your good work.
Virtual City are based in Nairobi and supply mobile solutions to the supply chain and agribusiness industry in Africa. African solutions for African problems. I got involved with them about 2 years ago to help them improve the quality of their products and I have been working on and off with them since then. Its been a bit of a journey we are getting there.
We have a number of client applications including android and wpf working in an online/offline mode over mobile networks. We need to process large amounts of incoming commands from these applications. These commands are also routed via the server to other client apps.
The application had originally used MVC and SQL server to synchronously process and store the commands but we were running into severe performance problems. We looked at various MQ solutions and decided to use RabbitMQ, WebApi & Mongo to improve processing throughput. While researching a .Net API for RabbitMQ I noticed that you had created the EasyNetQ API.
EasyNetQ greatly simplifies interacting with RabbitMQ and providing your needs are not too complicated you really don't need to know too much about the guts of RabbitMQ. We replaced the existing server setup in about a week. The use of RabbitMQ has greatly increased the scalability of the product and allows us to either scale up or scale out.
We are also using the EasyNetQ management API for monitoring queue activity on our customer services dashboard.
Kind Regards
Anthony Moloney

One of the great rewards of running an open source project is hearing about the fascinating ways that it’s used around the world. I really like that it’s an ‘African solution for African problems’ and built by a Kenyan development team. It’s also interesting that they’ve used OSS projects like RabbitMQ and Mongo alongside .NET. It reminds me of the Stack Overflow architecture, a .NET core surrounded by OSS infrastructure.