Dancing Elephants - Efficiently Working with Object Stores from Apache Spark and Apache Hive

Editor's Notes

• #5: key architectural considerations for improving Hadoop on the cloud Shared Data & Storage – the shared-data-lake is on cloud storage, it is not HDFS. Also memory and local storage play a different role – that of caching An import distinction in the cloud is On-Demand Ephemeral Workloads – this changes a number of things in fundamental ways. Shared Metadata, Security, and Governance remains important but need to be adjusted in the face of ephemeral clusters. And finally, Elastic Resource Management We need to shift our thinking away from cluster resource management and more towards SLA-driven workloads
• #6: Shared data requires a shared approach to metadata, security and governance. - Each ephemeral cluster cannot have it private copy of the metadata.. In cloud, metadata must be centrally stored - across all ephemeral clusters. The Metadata is not just the classic Hive metadata that describes the tabular data, about storing and tracking the lineage and provenance of data, - about details related to data pipeline processing and job management. Also, as data is ingested and processed, metadata needs to be created and adjusted Governance and securing the data remain critical and its matadata needs to be managed across all workloads. - Projects such as Ranger and Atlas need need to be evolved to fit the cloud environment.
• #9: on prem = 1, azure @ 3, AWS @ 2 There are also some problems in using S3 to replace HDFS esp for high perfm apps . . For example, HDFS provides much larger read/write throughput, and provides strong file system guarantees. We will talk about the practical problems shortly. Actually, they have different pros and cons for difference use cases, and fortunately they are not mutually exclusive. We suggest our customers use the right storage at the right place. But goal is move to the right one.
• #10: In all the examples, object stores take a role which replaces HDFS. But this is dangerous, because...
• #11: This is one of the simplest deployments in cloud: scheduled/dynamic ETL. Incoming data sources saving to an object store; spark cluster brought up for ETL Either direct cleanup/filter or multistep operations, but either way: an ETL pipeline. HDFS on the VMs for transient storage, the object store used as the destination for data —now in a more efficient format such as ORC or Parquet
• #12: Notebooks on demand. ; it talks to spark in cloud which then does the work against external and internal data; Your notebook itself can be saved to the object store, for persistence and sharing.
• #13: Example: streaming on Azure + on LHS add streaming
• #14: In all the examples, object stores take a role which replaces HDFS. But this is dangerous, because...
• #15: This picture is classic HDFS – 3 replicas and metadata Here in the rename all you expect is that the data remains and simply some metadata change – that is what happens in HDFS but no so in S3 Rename is important – task level and job level, Tasks may fail, job may fail, job speculation …
• #16: First off all there are no real directories in S3 – the directory names are part of the file name – Cloud object store does not treat “/” as a special separator. Note the hash function – if you rename the hash function result changes – hence rename or output commitment is not simple. There is no rename operation – a copy and delete has to be done The copied data is new (Yellow) The deleted data (Orange) So you might imagine – lets write directly to overcome the copy and delete – many committers did this – was great for the benchmark !!! EMR did this, Spark/Databricks did this …. All soon realized that in production loads you run into problems.
• #18: You might say – big deal the data in pending is deleted later – but image if you did direct output connector and was deleting failed task output Also same problem can occur for new data you added.
• #20: We will come back to the inconsistency problem and the new S3Guard
• #21: Everything uses the Hadoop APIs to talk to both HDFS, Hadoop Compatible Filesystems and object stores; the Hadoop FS API. There's actually two: the one with a clean split between client side and "driver side", and the older one which is a direct connect. Most use the latter. HDFS is "real" filesystem; WASB/Azure close enough. What is "real?". Best test: can support HBase.
• #22: This is the history of the connectors
• #24: Simple goal. Make ASF hadoop at home in cloud infra. It's always been a bit of a mixed bag, need to address many things: things fail differently. - Folks are surprised that this wasn’t the case – we did the s3 connectors in 2008, 2009, 2010!! - Netflix was the first real production workload – open sourced but integrated by community and not stack tested by community - EMR copied but kept proprietary Step 1: Azure. That's the work with Microsoft on wasb://; you can use Azure as a drop-in replacement for HDFS in Azure Step 2: EMR. More specifically, have the ASF Hadoop codebase get better performance than EMR
• #29: So what are the kinds of problems one can run into. Initially we did a lot of performance analysis, benchmarks and look at bug reports which showed consistency problems. We optimized many non-io-metaDataOperation – note these optimizations were not critical in HDFS since HDFS’s metadata and metadata operations are very cheap and no one bothered about optimizing them Work on the readPath and then on the writePath. Further work in query partitioning – which I will not talk abiut in this talk, The the output commitment and consistency problem – I will touch on this.
• #30: Here's a flamegraph of LLAP (single node) with AWS+HDC for a set of TPC-DS queries at 200 GB scale; we should stick this up online Shows number of operations (not time) – Small number of operations are IO, rest are metadata operations which are expensive REST operations. - getListing, seek, etc. only about 2% of time (optimized code) is doing S3 IO. Something at start partitioning data
• #32: Why so much seeking? It's the default implementation of read
• #34: things you will want for Hadoop 2.8. They are in HDP 2.5, and HDP 2.6 possible in the next CDH release. 1st two reduce cost of seeking. Seeking was originally implementing as reopening the connection and reading from the seeked position, However it sometimes cheaper to readahead , The Readahead config variable indicates that hundreds of KB can be skipped before reconnecting (yes, it can take that long to reconnect). The experimental fadvise random feature speeds reading in optimized binary formats like ORC which involves lots of random read, Indeed read at footer at the end of the file to then read the actual data from different portions of the file. The last deals supports fast uploads of data. There are parameters for buffering output as local files and then doing , multi-part uploads which offers the potential of significantly more effective use of bandwidth New version in Hadoop 2.8 is an improvement over the previous one that used to buffer in memory which used to conflict with with RDD caching..
• #35: A big killer turned out to be the fact that if we had to break and re-open the connection, on a large file this would be done by closing the TCP connection and opening a new one. The fix: ask for data in smaller blocks; the max of (requested-length, min-request-len). Result: significantly lower cost back-in-file seeking and in very-long-distance forward seeks, at the expense of an increased cost in end-to-reads of a file (gzip, csv). It's an experimental option for this reason; I think I'd like to make it an API call that libs like parquet & orc can explicitly request on their IO: it should apply to all blobstores
• #36: If you look at what we've done, much of it (credit to Rajesh & Chris) is minimizing HTTP requests and # of connections. Recall metadata operations are cheap in HDFS BUT NOT in S3. Each one can take hundreds of millis, sometimes even seconds due to load balancer issues (tip: reduce DNS TTL on your clients to <30s). A lot of the work internal to S3A was culling those getFileStatus() calls by (a) caching results, (b) perhaps not needed. Example: cheaper to issue a DELETE listing all parent paths than actually looking to see if they exist, wait for the response, and then delete them. The one at the end, HADOOP-13208, replaces a slow recursive tree walk (many status, many list) with a flat listing of all objects in a tree. This works only for the listStatus(path, true) call Better to use that API call, not do your own treewalk.
• #38: Note: much work remains: Data layout in cloud storage and its implications on applications needs further study - we need to understand the implications of sharding and throttling in S3. What we do know is that deep/shallow trees are pathological for recursive treewalks, and they end up storing data in the same s3 nodes, so throttling adjacent requests. Now lets talk about caching!
• #39: This benchmark shows the benefits of caching especially wrt to slow cloud storage. LLAP is next latest hive: . 2 benefits (both on-prem and esp cloud) thread-based work allocation (instead of process) and also caching. LLAP caches columnar data – only the columns that you need and also in an intermediate form. 26x faster!!!
• #43: Spark, early on focused on the direct outpuyt commiter ignoring the problem of atomicity and the careful design of the Hadoop’s HDFS centric file output committer, Performance was of good and the benchmarks and it helped Spark’s adoption. But the production problems surface and the direct output committer was removed, It was taken away because it can corrupt your data, without you noticiing. This is generally considered harmful.
• #44: Lets now turn out attention tp Netflix’s output connector, Output to local disk It uses a multi-part put to upload to S3, with a final output to commit the job. Significantly narrows the window of visibility of partial data and failure of job committer.
• #46: Recap – S3 is eventually consistent Additonal work we are doing to use Cloud DB to cache the metadata to achieve consistency.
• #48: Lets look at this in a little detail there is an additional metastore (e.g. Dynamo DB) where all metadata updates are stored. So an additional operation but some metadata operations are faster However, note there are issues about out-of-band, and source of truth that need to be considered.
• #55: And this is the big one, as it spans the lot: Hadoop's own code (so far: distcp), Spark, Hive, Flink, related tooling. If we can't speed up the object stores, we can tune the apps
• #59: Hadoop 2.8 adds a lot of control here. (credit: Netfllx, + later us & cloudera) -You can define a list of credential providers to use; the default is simple, env, instance, but you can add temporary and anonymous, choose which are unsupported, etc. -passwords/secrets can be encrypted in hadoop credential files stored locally, HDFS -IAM auth is what EC2 VMs need