Leveraging Endpoint Flexibility in Data-Intensive Clusters

Editor's Notes

• #2: Good afternoon. I’m … Today, I’m going to talk about network transfers that do not have fixed destinations. This is a joint work with … Written @ Berkeley (SIGCOMM) and presented on last August in Hong Kong
• #5: How it started: internet companies …. Main motivation in addition to regular stuff: Lower cost Less time Greater flexibility Linear scalability But How? And what happens that allows it?
• #6: Open source Started in Google…
• #7: Gordon Moore 1965
• #8: Capacity has increased while price has decreased
• #33: They analyze data in FB and Bing. Found out 33%.... ----------------- Many data-intensive jobs depend on communication for faster end-to-end completion time. For example, in one of our earlier works, we found that typical jobs at Facebook spend up to a third of their running time in shuffle or intermediate data transfers. As in-memory systems proliferate and disks are removed from the I/O pipeline, the network is likely to be the primary bottleneck. But what do the network usage of data-intensive clusters look like and where do they come from? To better understand the problem, we have analyzed traces from two data-intensive production clusters at Facebook and Microsoft. 1. Managing Data Transfers in Computer Clusters with Orchestra, SIGCOMM’2011
• #34: We have found something very interesting. While there has been a LOT of attention into deceasing reads over the network or managing intermediate communication, DFS replication creates almost half of all cross-rack traffic. Note that this doesn’t mean everyone was wrong; communication of intermediate data or shuffle is still a major source of job-level communication. But, the sources of these writes are ingestion of new data into the cluster and preprocessing of existing data for later use. Both of which do not show up when someone looks only at the jobs. Very small amount is actually created by typical jobs. We’ve also found that during ingestion many writers spend up to 90% of their time in writing. Well, that is their job. What is this DFS?
• #35: Distributed file systems are ubiquitous in data-intensive clusters and form the narrow waist. Diverse computing frameworks read from and write to the same DFS. Examples include GFS, HDFS, Cosmos etc. Typically, distributed file systems store data as files. Each file is divided into large blocks. Typical size of a block would be 256MB. Each block of a file is then replicated to three different machines for fault tolerance. These three machines are located in two different fault domains, typically racks, for partition tolerance. Finally, replicas are placed uniformly randomly throughout the cluster to avoid storage imbalance. Writes to a DFS are typically synchronous.
• #36: We address the traffic of distributed file systems in modern clusters like any other elephant flows in the network. We assume that the endpoints are fixed. All the existing work balance the network after the locations of the sources and destinations have already been decided. Because sources and destinations are fixed, they try to find different paths between them or try to change rates in different paths. But we can do more. Let us revisit the requirements of replica placement.
• #37: Notice that, as long as these constraints are met, the DFS does not care where actually the replicas have been placed. This means, we can effectively change the destinations of all replication traffic if we satisfy the constraints. We refer to such transfers as constrained anycast in that replicas can go anywhere, but they are constrained.
• #38: In this work, we present Sinbad. By steering replication traffic away from congested hotspots in the network Sinbad can improve the performance of the network. However, this can only be useful only if we have significant hotspot activities in the cluster.
• #39: We refer to this as the distributed writing problem. Given blocks of different size and links of different capacities, Sinbad must place the replicas in a way to minimize the average block write time as well as the average file write time. Note that, block can have different size because blocks are not padded in a DFS. Now for each block, we consider a job of that length and for each link we consider a machine of that capacity, we see that the distributed writing problem is similar to the job shop scheduling problem. And it is NP-hard.
• #40: Let’s take an example. We have the same network as before. We are going to assume that the three core-to-rack links are the possible bottlenecks. Replica placement requests from two different files come online. The black file has two blocks and the orange one has three. Now, let us assume that time is divided into discrete intervals. We must decide on the three requests during time interval T. We are also going to assume that intervals are independent; i.e., placement decisions during T will not affect the ones during T+1. Finally, we are going to assume that link utilizations are stable for the duration of replication or during T, and all blocks have the same size. It is clear that we should pick the least-loaded link because that will finish the fastest. Because all blocks are of the same size, it doesn’t matter which block we choose for minimizing the average block write time. If we also care about minimizing the file write times, we should always pick the smallest file (the one with the least remaining blocks) to go through the fattest link. Under these assumptions, greedy placement is optimal.
• #43: We propose a simple two-step greedy placement policy. At any point, we pick the least-loaded link and then
• #44: That brings us to Sinbad. Sinbad performs network-aware replica placement for DFS. <EXPLAIN> // Mention master-slave architecture etc.
• #45: That brings us to Sinbad. Given a replica placement request, the master greedily places it and returns back the locations. It also adds some hysteresis to avoid placing too many replicas in the same rack. Further details on the process can be found in the paper. One thing to note is that the interface is incredibly simple, which makes it all the more deployable. All in all, we needed to change only a couple hundred lines of code to implement the whole thing.
• #48: We have implemented Sinbad in the HDFS which is the de factor open source DFS used by traditional frameworks like Hadoop as well as upcoming systems like Spark. We have also performed flow-level simulation of the 3000-node facebook cluster. The three high-level question one might ask are — Does it improve performance? Does it improve network balance? Will the storage remain balanced. The short answer to all three is YES.
• #49: We have implemented Sinbad in the HDFS which is the de factor open source DFS used by traditional frameworks like Hadoop as well as upcoming systems like Spark. We have also performed flow-level simulation of the 3000-node facebook cluster. The three high-level question one might ask are — Does it improve performance? Does it improve network balance? Will the storage remain balanced. The short answer to all three is YES.
• #50: We consider performance from the perspective of the user (i.e., job performance) and that of the system (DFS performance) <EXPLAIN results> We have found that if we applied similar technique to in-memory storage systems like Tachyon, the improvements can be even higher because disks are never the bottlenecks. So, network-balance improved and performance improved as well. Upper bound: 1.89X
• #51: ציר – מקדם השונות של ה UTILIZATION We’ve found that network is highly imbalanced in both clusters. We are looking at a CDF of imbalance in core-to-rack downlinks in the facebook cluster. In the x-axis we have imbalance measured by the coefficient of variation of link utilizations. Coefficient of variation is the ratio of standard deviation to the mean of some samples, which is zero when all samples are the same, i.e., there is NO imbalance. In general, smaller CoV means smaller imbalance. We’ve measured link utilization as the average of 10s bins. We see that it is almost never zero and more than 50% of the time it is more than 1 (which is a typical threshold for high imbalance) Same is true for the Bing cluster as well. Given that a large fraction of traffic allow flexibility in endpoint placement and the network indeed has hotspots, we can now formally define the problem Sinbad is trying to address. ------------------------------------------------------------------------------------------------------------------------------ The network became more balanced as well. Notice that in both EC2 experiments and trace-based simulations, the orange moved toward the left, which indicate decreased network imbalance.
• #52: Sinbad optimize to Network – 10s check, decide where to put replicas by network and not only by Storage. Short term = 1h
• #55: There have been a LOT of work on better optimizing the network. And the solutions largely fall into three categories. The first approach is to increase the capacity of the network. This includes moving from 1GigE to 10GigE links and increasing bisection bandwidth of datacenter networks. In fact, there have been a lot of proposals on designing full bisection bandwidth networks. However, full bisection bandwidth does not mean infinite bandwidth, and the size of workload is always increasing. In practice, many clusters still have some amount of oversubscription in their core-to-rack links. The next approach is decreasing the amount of network traffic. All the work on data locality, and there have been many, try to decrease network communication by moving computation closer to its input. Recently, many researchers have looked into static analysis of data-intensive applications to decrease communication. These are all best effort mechanisms, and there is always some data that must traverse the network. This brings us to the third approach, that is load balancing the network. Typically it focuses on managing large flows and optimizing communication of intermediate data. Our recent work on Orchestra and Coflow also fall in this category. This work is about going one step further in this direction.