Spark Pipelines in the Cloud with Alluxio by Bin Fan

Editor's Notes

• #11: https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
• #12: https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
• #13: https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
• #14: https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
• #16: First I want to explain how Alluxio consolidate memory. Normally with Spark, different jobs have separate storage blocks. This causes data duplication which reduces the effective amount of memory. In addition, fetching the data is usually expensive and that is also doubled.
• #17: By using Alluxio, we reduce the number of data fetches from the slow storage and consolidate the memory used. As an additional benefit, Spark executors can be run with much less memory than before, reducing the chances of GC issues or OOM errors.
• #18: Even if there is only one Spark job, Alluxio still provides benefits in the case of crashes. Although crashes are never intended, in the big data world, we design architectures for them since they are inevitable.
• #20: When Spark crashes, both the storage and compute are gone.
• #21: With Alluxio in the picture, Spark crashes no longer cause data to be lost from memory.
• #23: Now let’s talk about how to write our Spark applications to leverage Alluxio. It is fairly straightforward. The interface is a file system interface, so reading and writing data is simply reading and writing Alluxio files.
• #24: For RDDs, it translates to these APIs. It is recommended to use text files when possible as that will get you the best performance. I will talk more about these two APIs in the performance evaluation portion.
• #25: Spark also provides APIs for more structured data in the form of dataframes. It is equally simple to read and write them with Alluxio, simply replace the original path with an Alluxio path.
• #27: We ran this experiment when Spark 2.0 came out with the latest version of Alluxio at the time. Since then we have kept track of improvements to Spark and Alluxio, but have not seen enough of a difference to rerun this performance evaluation. We compare several storage types in Spark with not storing the data in Spark but in Alluxio instead.
• #28: Let me explain what this graph is presenting. We first tested with a cached RDD, meaning the data was already in Spark or Alluxio, and we varied the size of the RDD and recorded how much time it took to run a scan on the RDD. The disk only line in green is as expected, much worse than all the others which are using memory. A more interesting observation is the performance difference between the Alluxio text file and object file. Text file is almost strictly better, this is because of the object serialization overhead, the file itself is all plaintext so using an object file was not helpful. I would recommend to use text file when possible. Using spark’s memory only is the best for small files, but abruptly performs worse once the data cannot be completely cached in memory. This happens as well for mem_only_ser the purlple line, but much last because of the small size of serialized objects. Alluxio scales linearly throughout the test and actually outperforms Spark caches for a single task after 32 GB or so.
• #29: The previous comparison was if the data was on SSD, which is still a relatively fast storage. If we instead put the data in S3, we see a much larger speed up. This is more representative of architectures because this allows compute and storage to be decoupled. In this case, we see 16x speed up even in this simple job, which is similar to some of the performance use cases I previously presented.
• #30: We also did a similar test with a parquet file using the data frame API. Here we did a simple aggregation which would access all the rows. The behavior is the same as before, using Spark’s native caching has an abrupt turning point where the performance degrades. These tests are run with default spark configurations which they suggest not to change. If we optimize for additional storage, we can move the point of bend, but it would still be present.
• #31: This is average of 7 runs. Range of s3 is 1132.765125 Range of Alluxio is 10.5890684 We also ran the same example against S3, and the variation in the test was fairly large. This is similar to what some users see in their storage either due to the storage itself, or their workload and sometimes both. Alluxio performance is much more consistent and provided on average 10x and up to 17x performance improvement.
• #33: With Alluxio in the picture, Spark crashes no longer cause data to be lost from memory.
• #34: Easy to use with spark