More Datacenters, More Problems

Editor's Notes

• #4: So who am I, and why am I qualified to stand up here? I am a member of the Data Infrastructure Streaming SRE team at LinkedIn. We’re responsible for Kafka and Zookeeper operations, as well as Samza and a couple iterations of our change capture systems. SRE stands for Site Reliability Engineering. Many of you, like myself before I started in this role, may not be familiar with the title. SRE combines several roles that fit together into one Operations position Foremost, we are administrators. We manage all of the systems in our area We are also architects. We do capacity planning for our deployments, plan out our infrastructure in new datacenters, and make sure all the pieces fit together And we are also developers. We identify tools we need, both to make our jobs easier and to keep our users happy, and we write and maintain them. At the end of the day, our job is to keep the site running, always.
• #6: These are the numbers I presented this time last year, as far as how much data we push around in Kafka at LinkedIn. Over the last year, it’s changed significantly We now have well over 1800 brokers in total in our 80+ clusters Which are managing over 95,000 topics With over 1.3 million partitions between them, not including replication We’ve gone from 875 billion messages a day to over 1.3 trillion, and that was a slow day There is now over 330 terabytes per day flowing into Kafka, almost double And consumers are reading over 1.2 petabytes per day out. Of course, those are both compressed data numbers I’m changing the peak load numbers, choosing to show what happens in our single largest production cluster. At peak, we’re receiving over 2 million messages per second For a total of 4.7 gigabits per second of inbound traffic And the consumers are reading over 15 gigabits per second at the same time Again, this is compressed data, and these clusters are held at a max of 60% of disk and network utilization. This is a fairly astonishing growth rate for the amount of data we are moving around with Kafka. Some of it comes from standing up new datacenters, so let’s move directly into what that looks like.
• #7: What are the things we are going to cover in this talk? We’ll start by talking about the basics of how Kafka works, very briefly, and move right into what a tiered architecture looks like along with the infrastructure tool we use for creating our tiers – mirror maker. We’ll then step back and discuss the reasons why we run multiple datacenters, and what that means for our design choices. I will cover performance tuning, specifically when it comes to laying out and managing tiered clusters. Lastly, we’ll talk about what work is going on right now that will continue to improve the ecosystem for running large Kafka installations, and what you can do to get involved.
• #9: I won’t be going into too much detail on how Kafka works. If you do not have a basic understanding of Kafka itself, I suggest checking out some of the resources listed in the Reference slides at the end of this deck. Here’s what a single Kafka cluster looks like at LinkedIn. I’ll get into some details on the TrackerProducer/TrackerConsumer components later, but they are internal libraries that wrap the open source Kafka producer and consumer components and integrate with our schema registry and our monitoring systems. Every cluster has multiple Kafka brokers, storing their metadata in a Zookeeper ensemble. We have producers sending messages in, and consumers reading messages out. At the present time, our consumers talk to Zookeeper as well and everything works well. In LinkedIn’s environment, all of these components live in the same datacenter, in the same network. What happens when you have two sites to deal with?
• #10: Multiple datacenters is where this starts to get interesting. Here is an example of a layout that uses one Kafka cluster. We’re keeping the cluster in a single datacenter, because having it span datacenters is an entirely different level of complexity. In addition, Kafka has no provision for reading from the follower brokers, so you would still be crossing datacenters with your clients. The problem with this layout should be quite obvious – if we lose Datacenter A, we’ve lost everything. Not only do we have concerns with network partitions between the datacenters cutting off access for one consumer or producer or another, we have no redundancy at all.
• #11: Another single cluster alternative is to have the cluster span all the datacenters. That is, at least a couple brokers in each datacenter, all sharing the same Zookeeper instance and operating as one cluster. This layout has lots of problems as well, though. First, you’re still dealing with network latency across sites. This is compounded by spreading out the Zookeeper ensemble (and you have to, otherwise you can’t tolerate the loss of the datacenter that the ensemble is in). And your client connections are not a simple as pictured above. They actually cross all over the datacenters and connect wherever the leaders are. The other big problem is that you need to make sure that all partitions have replicas that are in separate datacenters. Brand new in 0.10 is the ability to have the brokers do rack-aware replica assignment, and you can utilize this to make sure the initial replica assignments are correct. Of course, that’s only the initial assignment – if you add more brokers or more datacenters and want to rebalance partitions, you’re on your own. Also, it’s brand new code, and if you’re anything like me you’re wary of relying on it just yet.
• #12: So to improve this situation, we’ll run a Kafka cluster in each of our primary datacenters, A and B. In this layout, C is a lower-tier datacenter where we don’t have producers of the data, only consumers. Consider it a backend environment where you run things like Hadoop. Our producers all talk to the local Kafka cluster. Consumers in the primary datacenters talk to their local cluster as well. Now if we lose either datacenter A or B, the other datacenter can continue to operate. We’ve pushed more complexity on the consumers that need to access all of the data from both datacenters, however. They have to maintain consumer connections to both clusters, and they will have to deal with networking problems that come up as a result. In addition, latency in the network connection can manifest in strange ways in an application.
• #13: Now we iterate on the architecture one more time. We add the concept of an aggregate Kafka cluster, which contains all of the messages from each of the primary datacenter local clusters. We also have a copy of this cluster in the secondary datacenter, C, for consumers there to access. We still have cross-datacenter traffic – that can’t be avoided if we need to move data around. But we have isolated it to one application, mirror maker, which we can monitor and assure works properly. This is a better situation than needing to have each consumer worry about it for themselves. We’ve definitely added complexity here, but it serves a purpose. By having the infrastructure be a little more complex, we simplify the usage of Kafka for our customers. Producers know that if they send messages to their local cluster, it will show up in the appropriate places without additional work on their part. Consumers can select which view of the data they need, and have assurances that they will see everything that is produced. The intricacies of how the data gets moved around are left to people like me, who run the Kafka infrastructure itself.
• #14: We’ve chosen to keep all of our clients local to the clusters and use a tiered architecture due to several major concerns. The primary concern is around the networking itself. Kafka enables multiple consumers to read the same topic, which means if we are reading remotely, we are copying messages over expensive inter-datacenter connections multiple times. We also have to handle problems like network partitioning in every client. Granted, you can have a partition even within a single datacenter, but it happens much more frequently when you are dealing with large distances. There’s also the concern of latency in connections – distance increases latency. Latency can cause interesting problems in client applications, and I like life to be boring. There are also security concerns around talking across datacenters. If we keep all of our clients local, we do not have to worry about ACL problems between the clients and the brokers (and Zookeeper as well). We can also deal with the problem of encrypting data in transit much more easily. This is one problem we have not worried about as much, but it is becoming a big concern now. The last concern is over resource usage. Everything at LinkedIn talks to Kafka, and a problem that takes out a production cluster is a major event. It could mean we have to shift traffic out of the datacenter until we resolve it, or it could result in inconsistent behavior in applications. Any application could overwhelm a cluster, but there are some, such as applications that run in Hadoop, that are more prone to this. By keeping those clients talking to a cluster that is separate from the front end, we mitigate resource contention.
• #17: Why do we even want to have multiple datacenters in the first place? The answer may be obvious to some, but it’s always worth starting out with a clear statement of what we’re hoping to accomplish. This makes it easy to make decisions about how the architecture should look to solve the problem that we intend to solve. The single largest reason behind having multiple sites is for disaster recovery. If one site is down for some reason, whether that is due to a loss of power, a natural disaster, or zombies, we want to be able to continue operations. Hopefully in an uninterrupted manner. This means that we need the entire stack of applications represented in all datacenters. We also need enough capacity to handle the failure scenario we are planning for. If you have 2 datacenters, you want to plan for losing one. If you have 3 datacenters, you might still only plan for the loss of one, which means you don’t have to have 300% of your planned capacity out there. Another big reason for multiple sites is to get the content as close to the users as possible. A round-trip for data that has to go halfway around the world is at least half a second, and if you have a few of those in a page load, your users are going to have a bad time. There are a lot of ways to accomplish this goal, from using a ready-made content distribution network, to setting up points of presence that terminate and aggregate connections, to having full-blown datacenters in strategic locations. A sometimes overlooked reason for multiple datacenters, especially for those of us with our heads down in the code, is legal and/or political reasons. There can be tax advantages to having datacenters in certain countries. There may also be legal requirements for serving or storing a local user’s content locally. You may think about China here, but I think more about countries like Germany that have strong federal data protection laws.
• #18: Disaster recovery is the first place we look – if we want to get big, we have to be able to tolerate problems. It gets much easier to run more than one datacenter when all of them look exactly the same. Run all the same services everywhere, because special little snowflakes are difficult to move around and fail over. This can be difficult in practice, however, so we need to make accommodations for them. We call these “single master” services – there can be only one of them within our infrastructure. Services like this will need to coordinate, because in order to handle a disaster you’ll need to have a second copy of them somewhere, whether as a warm or cold standby. This requires cross-site coordination, and one way to do this is to use Zookeeper. Keep in mind that your Zookeeper ensemble needs to live in at least 3 sites to make this work. So if you have two datacenters, you’ll need a third site to act as an arbiter. Another problem with this type of application is that you have to fail over your position from one Kafka cluster to another. As offsets are different between clusters, this is a little complex to do. We currently have the ability to request an offset from the broker given a timestamp, but you probably won’t get what you want. Kafka satisfies this request by giving you the first offset of the log segment that your timestamp falls into by looking at the modification time of the log segments. This means that the offset you get is only guaranteed to be before the timestamp you asked for. It could be seconds, minutes, hours, or days before, but it will be before. Thankfully, there’s a fix coming for this. KIP-33, worked on as a set of changes by Becket, one of LinkedIn’s developers, adds a time-based index. This will allow Kafka to return a much more granular offset as an answer to that query. You may ask me “what about running in Amazon Web Services?” The answers are all the same, it’s just the language that is different. Rather than speaking about datacenters, we talk about “availability zones”.
• #19: But nobody likes thinking about disasters, so let’s plan for the best of all worlds – you have so much traffic you need more capacity and you want to spread it out so that your users get the best experience. It’s actually not that much different from disaster planning. You’re still working with multiple sites, but you’re probably going to add more over time. So where you really only need two sites for DR, you might have 3, 4, or dozens of sites for geolocalization. One thing to be concerned about here is where you have aggregate Kafka clusters. Every copy of your messages is going to cost you money – both in storage and bandwidth. A good plan here is to designate a limited number of datacenters (2 is a good number, remembering DR) where aggregate data lives. This could be two “back office” type sites, or it could be two of your production datacenters that specifically set up for this. Run the applications that need aggregate data there, and then push the results out to applications in the other sites to work with. This is a great place to run stream processing in a framework like Samza, where you are performing online transformation of your messages and generating new data for applications. Again, when it comes to AWS and Google Cloud, and the like, you’re just dealing with a difference in language. When you want to do geolocalization, you’re most likely going to talk about using different regions. But keep an eye on those data transfer costs!
• #20: Legal requirements for data storage can present a lot of difficulties for us. We may start out wanting to make all the data available everywhere, but the reality is that what we really want is to make sure the right data is available where it is needed. What the “right data” is can be heavily influenced by the legal and political concerns we just mentioned. For example, what happens if we store a US user’s data in China? Or the opposite? If we have a datacenter in Germany, is there a problem a German citizen’s data is stored outside the country? Do we need to handle it differently? For the most part, we only want to deal in whole topics. If we want to mirror messages from one Kafka cluster to another, it’s much easier for us to just mirror the entire topic. If it’s required that we filter the messages as we mirror them, a significant layer of complexity has been added. We now have to inspect every message and make a decision. It also complicates monitoring systems like audit (which I will talk about later) So what is the best practice? Have enough topics so that your consumers can operate without throwing away messages. This means that you should try to avoid having consumers that filter messages – you should be able to accomplish that by only consuming one of several topics.
• #21: Does anyone know who this woman is? Her name is Helen Dixon, and she’s the Irish Data Protection Commissioner. Like many other companies, we hold ourselves to the standards of the IDPC for protecting our members’ data. There are specific rules about how long you can retain certain types of data, and how quickly it must be removed if requested. This leads us to our first best practice when it comes to message retention – only keep it around as long as you have to. This is not only important from a protection and privacy point of view, but from a resource point of view as well. The more data we retain, the more storage we need to have online. Fast storage is expensive, and we’re already keeping multiple replicas within a cluster. We also need to consider security – the more data we have, the more vulnerable we are making ourselves. This leads us to another good idea – keep the data away from the front lines of your infrastructure as much and as quickly as you can. The further back into your infrastructure your data is, the more layers an attacker would have to go through to get to it, the less vulnerable that data is. This means that if we can, we should have lower retention times in the first level clusters, relying on either other Kafka clusters (that are deeper into our infrastructure) or other systems entirely (like HDFS) to store messages for longer periods of time.
• #22: More components means that we have more places to poke and prod to get the most efficiency out of our system. With multiple tiers most of this revolves around making sure the sizes of everything are correct.
• #23: This is as good a time as any for a little self-promotion. Many of the questions around how to set up and lay out Kafka clusters, including specific performance concerns and tuning, are covered in this fine book that I am co-authoring. You’ll also find a trove of information about client development, stream processing, and a variety of use cases for Kafka. We currently have 4 chapters complete, and it’s available from O’Reilly under their early access program. We expect to have the book completed late this year, or early next, with chapters being released as soon as we can write them.
• #24: It may not strictly be tuning, but having your Kafka clusters be the right size is the first place you need to start. It can be a little difficult to determine exactly how large your cluster should be, but there are a few major points you have to consider. First, how much disk space do you have on each broker, and how much space do you need to maintain message retention? One of our rules is that we keep disk usage for the log segments partition to under 60%. This allows us enough headroom to move partitions around when needed (especially because retention for the partition resets when you move it). The next concern is how much network you have. If you have gigabit network interfaces, and your Kafka cluster is going to receive 5 gigabits per second of traffic at peak, you need to take that into account. CPU, memory, and disk I/O all take a back seat to these concerns, because they mostly drive how fast your cluster operates, not if it operates at all. Size your local clusters first, then you can consider how large your aggregate cluster needs to be. For the most part, taking the number of local clusters you have, multiplied by the number of brokers in each local cluster, will give you the size of your aggregate cluster. If you have 3 local clusters with 10 brokers each, your aggregate cluster should be at least 30 brokers. You’ll also need to take into account the number of consumers of the aggregate messages. This can affect how much network bandwidth you need, which will change the number of brokers that you need.
• #25: You will also need to size your topics appropriately. This tends to be a topic of much discussion, because there are so many variables that get considered here. For example: How many brokers do you have? Do you want to perfectly balance the topic across the brokers? If so, you should have then number of partitions be a multiple of the number of brokers How many consumers does your topic have in its largest consumer group? If you have 8 partitions and 16 consumers, 8 of those consumers will be sitting idle Does your application have specific requirements around partition counts? If you are using keyed messages, you may want to go with a larger number of partitions to start with so you don’t have to expand it later based on other criteria Another concern we have is around keeping the size of a partition on disk manageable. Very large partitions can be harder to keep balanced in a cluster to make sure each broker is doing its fair share of work. We use a guideline internally of making sure partitions do not exceed 50 gigabytes on disk. When they get close or exceed that, we expand the topic (provided it is not a keyed topic). Once again here, the partition counts in your aggregate cluster should be a simple calculation. For most topics, we take the number of partitions in the local cluster, multiply it by the number of local clusters, and that is the number of partitions in the aggregate cluster. You also want to check your partition counts regularly, especially if you use automatic topic creation. An imbalance in the number of partitions between clusters can bog down mirror maker very quickly. Another thing to consider is how much retention you need. There’s nothing that says that retention has to be the same in the local and aggregate tiers. You may want to retain messages longer in aggregate, keeping them in the local tier only long enough to get them out. Just remember that this can change your sizing calculation for the aggregate clusters. If you have twice the retention, you may very well need twice the number of brokers.
• #26: The answer of where to put the mirror makers, local to the source cluster or local to the target cluster, used to have a simple answer – putting it with the target cluster was safer. However, now that Kafka has introduced TLS communications, and we have a push to encrypt all cross-datacenter traffic, the game has changed. If we put the mirror maker in the datacenter with the produce cluster, we have a better layout from a network point of view. If a mirror maker fails to consume messages, it will just not commit offsets and you will not lose messages. You will just slow down or stop consuming. If the mirror maker fails to produce messages, then more likely than not it will start dropping messages. This means we want to keep network problems on the consumer side as much as possible. However, there’s a significant performance hit to consuming messages over TLS, due to the loss of a zero copy send in the broker. When producing messages over TLS, we don’t take this performance hit, so if you don’t have a directive to encrypt every link, even within your datacenter, placing the mirror makers in the datacenter with the consume cluster is now the way to go. Newer mirror maker versions, which exit on any produce failure, make this a little easier to deal with.
• #27: We introduced another component with Mirror Maker, so we also need to size that appropriately. When we talk about sizing, we are talking about the number of copies of mirror maker with the same consumer and producer configuration, which is one pipeline. With mirror maker, it’s mostly about network throughput. Because of the decompression and recompression of message batches, you’re probably never going to run at wire speed. This means that you can easily co-locate multiple mirror makers on one set of servers to efficiently use them. You should also make sure that you are running more copies of mirror maker than you need to handle your peak traffic. If you fall behind, such as if you have a network partition for a period of time, you want to be able to catch up quickly. If you don’t have excess capacity, it will take a long time, or you will just continue to fall behind. You should also run multiple consumer and producer streams in each copy of mirror maker, as this will allow you to take advantage of the parallel nature of having multiple partitions. If you can process 15 megabytes per second at peak on one stream, you won’t get 30 with two streams, but you’ll do a lot better. We run with 8 consumer streams and 4 producer streams, and it works out pretty well. We also co-locate up to 11 mirror makers on one host, each for a separate pipeline. There are a few other parameters to consider. One is the partition assignment strategy. We asked our developers to add a round robin strategy of balancing partitions for wildcard consumers, like mirror maker. This provides a nice balance of partitions across your mirror makers, and should almost certainly be the configuration you use. You should also set the number of in flight requests per connection. A higher number will make things go faster, but it will also mean more loss of messages if mirror maker breaks. The linger time for the producer is another thing to look at. A longer linger time will allow mirror maker to assemble more efficient batches of messages, but it will also mean that messages take a little longer to get through the pipeline. Weigh which tradeoffs are the right ones for you.
• #28: Another thing you can do is to provide separate paths for different topics between the same two clusters. We do this at LinkedIn because not all topics are created equal. We have high priority topics. For you, this could be topics that change search results. For us, it’s mostly topics that are used for hourly or daily reporting. Most other topics, especially headed to Hadoop, we’re OK if they’re a little delayed. But if the hourly report to the executives is delayed, you can be sure that I’m fielding phone calls as to exactly what is broken and when it will be fixed. Another type of topic you may want to separate out is low latency topics. This could be topics that are used for messaging, or things that generate responses to users. We don’t want to wait for these messages to batch up and send. For these topics, we have two separate mirror maker pipelines that run in parallel. The high priority mirror maker has a small whitelist of topics, and the other mirror maker has a blacklist that contains the same topic list. This way a bloated topic that is not considered a priority will not delay the most important topics. It also means that the priority mirror maker starts up faster, and takes less time to catch up when there is a problem. These are all very good things if it means the CEO doesn’t know my name.
• #30: There are a number of things that can be improved upon, both in the brokers and in the mirror maker, to make it easier to set up and manage multiple datacenters. In the architecture I laid out, and the one in use currently at LinkedIn, we have both local and aggregate clusters in every datacenter. This is wasteful, because we have two copies of the local cluster’s messages in a datacenter. If we were to use namespaces, we could have a single aggregate cluster that contains the messages for all datacenters, and clients could choose to consume either local or aggregate messages by using a wildcard. Namespaces could be accomplished either natively in Kafka (a KIP which has already come up and been rejected), or could be handled by using our wrapper library to prefix topic names. Another big problem is that we are using RAID and providing a single mount point to the Kafka brokers for a log dir. This is because there are some issues with the way JBOD is handled in the broker. Specifically, the brokers assign partitions to log dirs by round robin, not taking into account current size. In addition, there are no administrative functions to move partitions from one directory to another. And if a single disk fails, the entire broker fails. If JBOD was more robust, we could have replication factors of 3 or 4 without an increase in hardware cost, which would allow us to have “no data loss” configurations.
• #31: The big improvement to mirror maker is the creation of an identity mirror maker, which would keep message batches together in the exact same partition from source to target cluster. This would completely eliminate the compression overhead from the mirror maker, making it much faster and more efficient. Of course, this requires maintaining the partitions counts in the clusters properly, and allowing the mirror maker to increase partition counts in a target cluster if needed. Another issue with mirror maker is that as you add new local clusters, you geometrically increase the number of paths you have to mirror messages over. If each path is a mirror maker instance, this gets out of control very quickly. There’s no good solution for this right now – you used to be able to configure a mirror maker with multiple consumers, but not in 0.8 or later. Bringing it back might not be a bad idea as a way to reduce the infrastructure complexity.
• #32: That leads into the idea of multi-cluster management. While there are a couple people making some headway on this in the open source world, we still lack a solid interface for managing Kafka clusters as part of an overall infrastructure. This would include maintaining topic configurations across multiple clusters and easily configuring and visualizing the mirror maker links between them. Another piece needed is better client monitoring overall. Burrow provides us with a good view of what the consumers are doing, but there’s nothing available yet for producer client monitoring. We, of course, have our internal audit system for this. And other companies have their own versions as well. It would be nice to have an open source solution that anyone can use for assuring that the producers are working properly. We could also use better end-to-end monitoring of our Kafka clusters, so we can know that they are available. We have a lot of metrics that can track information about the individual components, but without a client view of the cluster, we don’t know if the cluster is actually available. We also have a hard time making sure that the entire pipeline is working properly. There’s not a lot available for this right now, but watch this space…
• #33: So how can you get more involved in the Kafka community? The most obvious answer is to go apache.kafka.org. From there you can Join the mailing lists, either on the development or the user side You’ll find people on the #apache-kafka channel on Freenode IRC if you have questions We also coordinate meetups for both Kafka and Samza in the Bay Area, with streaming if you are not local You can also dive into the source repository, and work on and contribute your own tools back. Kafka may be young, but it’s a critical piece of data infrastructure for many of us.