Strata EU tutorial - Architectural considerations for hadoop applications

Editor's Notes

• #25: Talk about confusion even with knowledgeable users about how all the components fit together to implement applications.
• #27: Think about kittens/cars getting challenged
• #29: Data ingestion – what requirements do we have for moving data into our processing flow? Data storage – what requirements do we have for the storage of data, both incoming raw data and processed data? Data processing – how do we need to process the data to meet our functional requirements? Workflow orchestration – how do we manage and monitor all the processing?
• #31: We have a farm of web servers – this could be tens of servers, or hundreds of servers, and each of these servers is generating multiple logs every day. This may just be a few GB per server, but the total log volume over time can quickly become terabytes of data. As traffic on our websites increases, we add more web servers, which means even more logs. We may also decide we need to bring in additional data sources, for example CRM data, or data stored in our operational data stores. Additionally, we may determine that there’s valuable data in Hadoop that we want to bring in to external data stores – for example info to enrich our customer records.
• #32: Add title slide for storage reqs
• #34: Data needs to be stored in it’s raw form with full fidelity. This allows us to reprocess the data based on changing or new requirements. Data needs to be stored in a format that facilitates access by data processing frameworks on Hadoop. Data needs to be compressed to reduce storage requirements.
• #39: So simple! We can just write a quick bash script, schedule it in cron and we are done. This is actually not a bad way to start a project – it shows value very quickly. The important part is to know when to ditch the script.
• #40: I typically ditch the script the moment additional requirements arrive. The first few are still simple enough in bash, but soon enough…
• #41: There’s a need for a more sophisticated approach, or we’ll be drowning in bash scripts
• #42: Even if we use an engine that allows for complex workflows like Spark – orcehstration makes things like recovering from error, managing dependencies and reusing components easier.
• #43: Now that we understand our requirements, we need to look at considerations for meeting these requirements, starting with data storage.
• #45: Now that we understand our requirements, we need to look at considerations for meeting these requirements, starting with data storage.
• #48: Random access to data doesn’t provide any benefit for our workloads, so HBase is not a good choice. We may later decide that HBase has a place in our architecture, but would add unnecessary complexity right now.
• #50: Recall that we’ll be dealing with both raw data being ingested from our web servers, as well as data that’s the output of processing. These two types of data will have different requirements and considerations. We’ll start by discussing the raw data.
• #51: We could store the logs as plain text. This is well supported by Hadoop, and will allow processing by all processing frameworks that run on Hadoop. This will quickly consume considerable storage in Hadoop though. This may also not be optimal for processing.
• #52: We could store the logs as plain text. This is well supported by Hadoop, and will allow processing by all processing frameworks that run on Hadoop. This will quickly consume considerable storage in Hadoop though. This may also not be optimal for processing.
• #55: SequenceFiles are well suited as a container for data stored in Hadoop, and were specifically designed to work with MapReduce. SequenceFiles provide Block compression, which will compress a block of records once they reach a specific size. Block level compression with sequence files allows us to use a non-splittable compression format like Gzip or Snappy, and make it splittable. Important to note that SequenceFile blocks refer to a block of records compressed within a SequenceFile, and are different than HDFS blocks. What’s not shown here is a sync marker that’s written before each block of data, which allows readers of the file to sync to block boundaries.
• #56: Avro can be seen as a more advanced SequenceFile Avro files store the metadata in the header using JSON. An important feature of Avro is that schemas can evolve, so the schema used to read the file doesn’t need to match the schema used to write the file. The Avro format is very compact, and also supports splittable compression.
• #61: Recall that much of our access to the processed data will be through analytical queries that need to access multiple rows, and often only select columns from those rows.
• #67: Access to /data often needs to be controlled, since it contains business critical data sets. Generally only automated processes write to this directory, and different business groups will have read access to only required sub-directories. /app will be used for things like artifacts required to run Oozie workflows,
• #68: Note that partitions are actually directories.
• #69: Note that partitions are actually directories.
• #70: Indicate this applies to raw and processed
• #75: Typically we are looking at few files landing at the FTP site once a day, scheduling a job to run on an edgenode of the cluster once a day, fetch the files and stream to HDFS is fine. If an import fails, it will fail completely and you can retry.
• #78: Ease of deploy and management is important Customer will not write code Interceptors are important Data-push is important Data will always end up in Hadoop
• #80: Many planned consumers High availability is critical You have control over sources of data You are happy to write producers yourself
• #83: Does not require programming.
• #86: Only without a debugger
• #87: Multiple agents acting as collectors provides reliability – if one node goes down we’ll still be able to ingest events. Flume provides support for load balancing such as round robin.
• #92: Does not require programming.
• #93: Does not require programming.
• #94: Does not require programming.
• #122: There’s a need for a more sophisticated approach, or we’ll be drowning in bash scripts
• #125: There’s a need for a more sophisticated approach, or we’ll be drowning in bash scripts
• #127: There’s a need for a more sophisticated approach, or we’ll be drowning in bash scripts
• #128: There’s a need for a more sophisticated approach, or we’ll be drowning in bash scripts
• #129: There’s a need for a more sophisticated approach, or we’ll be drowning in bash scripts
• #132: One or two hive actions would do, maybe some error handling The workflow is simple enough to work in any tool – Bash, Azkaban… but Oozie’s dataset triggers make it a good fit for this use-case Note that if we were to use Kafka, the workflow would be even simpler and we wouldn’t use time-based scheduling
• #133: There are a lot of Orchestration tools out there. And ETL tools typically do orchestration too. We want to focus on the open-source systems that were built to work with Hadoop – they were built to scale with the cluster without a single node as a bottleneck
• #138: No XML is huge for many people
• #140: One or two hive actions would do, maybe some error handling The workflow is simple enough to work in any tool – Bash, Azkaban… but Oozie’s dataset triggers make it a good fit for this use-case Oozie also makes recovery from errors easier: Data sets are immutable, actions are idempotent and oozie supports restarting workflow and running only the failed action.
• #141: This is something you’ll need to add yourself, possibly using an RDBMS and custom java action, if advanced metrics are important
• #153: We should probably do this with Hue