Producing Spark on YARN for ETL

Editor's Notes

• #2: Good afternoon everybody. I'm Ashwin and this is Nezih. We work for the big data platform team at Netflix. Today we are going to talk about our experience productionizing spark for running ETL workloads on YARN
• #3: I'm sure all of you have seen this netflix page before. Do you also know that spark contributes pipeline that does personalized ordering of the rows in this page. We will talk about this application in the second part of the presentation. Not just that, spark is also used to train some of machine learning models that powers our recommendations alogorithms.
• #4: Netflix is a data driven company collect a lot of data from user interactions, platforms, services One example where we use this data is A/B testing. Lets say we have UI change actually improves the experience The biggest challenge that we face…
• #5: Just to give an idea of at what scale we operate, I'm going to share with you some numbers.
• #6: this would generate a lot of data. But how much
• #8: Today, we'll walk you through three sections
• #9: We need to answer two questions. First ques how do we get data in platform? It looks like this
• #10: two pipelines. event data, so this is all the data from devices and microservices running out in the cloud. This goes into a kakfa clusters,ursula. This pipeline runs under a min. At the bottom, dimension data pipeline. live data in casssandra. Everyday this data is backed up and a tool called Aegisthus processes these backups and writes into our data warehouse. In the end, we have all of event and dimension data on s3, ready for processing.
• #11: At this point, our data is on s3 which is our source of truth, parquet .. At the very top in our interface. hundreds of internal users using platform. bigdataportal, one stop shop to access our big data services. From this portal, users pick a compute engine and run their jobs. cool features like schema browsing and accessing s3. The portal uses the Bigdata api under the hood..
• #12: Here is how our spark on yarn deployment looks like Thanks to genie, multiple spark versions. This helps experiment with new features, bug fixes To make best use of our resources, spark alongside mapreduce
• #14: Now lets move on to the technical challenges. We are going to walk through the lifecycle of a spark job, as we do that we'll talk about the challenges that we had.
• #15: resource manager, master process, managing the cluster resources. driver via the AM requests from initial set of executors from the RM, and NM launches In a typical spark job, probably use the RDD interface.. We faced two technical challenges here. the first one is a limitation of RDD interface and second one is perf optimization. T : Here is the first one
• #16: Recently we had a usecase where we wanted to merge small files. merges partitions according to a given number of paritions, but we want to merge by file size added support for passing a custom coalescer to this interface. and we implemented a size based coalescing strategy T : Now lets look at the optimization in UnionRDD.
• #17: We noticed select * from table limit 10, slow against partitioned hive table Found each hive partition, spark hadoopRDD,UnionRDD. reason poor performance sequential listing of parent hadoopRDD partitions. resolved this by computing the partitions of unionRDD in parallel. T : This fix was good but we found another important problem in s3 filesystem that was affecting the listing performance.
• #19: in hadoop we found, Filesystem making an unnecssary rpc call to s3, which is expensive. We fixed this issue by removing this redundant rpc call. This along with Union RDD fix improved in split calculation performance by 20x. T: : This problem was on the read path, we hit another issue on the write path, which was with Output committers.
• #20: Output committer writes the output of single task in an all or nothing fashion. Each task writes output to a temp directory. First successful.. But this doesn't work well with S3. To solve this we use our internal s3 output committer, which we plan to open source in the future.
• #22: At this point, driver has finished listing and it starts running tasks on the executors. dynamic allocation will kick in, and scale job's execturos up and down based on the workload. We had several major problems with dyn alloc and I'll now talk about two of them here. T : Here is the first one.
• #23: When dynamic allocation is enabled, we found that reading broadcast data can take a long time. reason is executors retrieve block locations only once from the driver during the read. But with dynamic allocation , replicas can be removed resulting in stale entries in the retrived block locations. Production job example We fixed this problem on the executor side, now the executors refresh the block locations from the driver after multiple failed attempts. T: The second problem we had is with locality optimization.
• #24: noticed a spark application failed due to timeouts. On debugging we found that AM would cancel and resend container requests if the one, locality preference is no longer needed or two if locality preference is not set. Problem is we run on s3 and s3 doesn't have locality information set, which was why we see this thrashing. We fixed this by not cancelling container requests that don't have a locality pref set.
• #29: See also https://issues.apache.org/jira/browse/SPARK-8890 Inserting a repartition minimizes the number of output files and the memory consumption. In progress, not merged.
• #32: Due to sequential processing one large application event log can block new applications from showing up. Other configs which helped fix OOM errors Move from CMS to G1GC Garbage collector -XX:NewRatio=1 (give more space to YoungGen)
• #41: Available in Spark 2.0