Driving Enterprise Data Governance for Big Data Systems through Apache Falcon

Editor's Notes

• #4: ET: Tactical POS Added benefits – consortium
• #5: Transition to Andrew
• #9: Transition to Andrew
• #10: Thanks Justin, Here are Falcon’s primary features. 1 The first is to manage the data lifecycle in one common place. 2 The second is to facilitate quick deployment of replication for business continuity and disaster recovery use cases. This includes monitoring and a base set of policies for replication and retention 3 Lastly, Falcon provide foundation audit and compliance features – visuallization and tracking of entity lineage and collection of audit logs
• #12: This is the high level Falcon Architecture Falcon runs as a standalone server as part of your Hadoop cluster A user creates entity specifications and submits to Falcon using the API Falcon validates and saves entity specifications to HDFS Falcon uses Oozie as its default scheduler Dashboard for entity viewing in Falcon UI Ambari integration for management
• #13: Feeds have location, replication schedule and retention policies Meta info including frequency, where data is coming from (source), where to replicate (target), how to long to retain
• #14: Let take a look at the Data Pipeline or workflow. ** read high level **
• #15: Hive – HQL scripts Pig scipts Oozie workflows
• #16: Once a pipeline is create you’ll want to run it. This means you probably want to monitoring as well. Falcon in conjunction with Ambari has centralized monitor ** bullets **
• #17: Ok let chat about Replication with Falcon – which is very efficient. In this example with a primary cluster with a typical workflow There is business requirement to replicate this to a Failover cluster ** builett **
• #18: Falcon has flexible data retention policies, it’s able to model the business compliance requirements. Sophisticated retention policies expressed in one place Simplify data retention for audit, compliance, or for data re-processing In this example, different dataset in a workflow can have different retention policies.
• #19: We realize at many type of workflow have inputs from different system with may be in different regions. Falcon has logic built-in to handle this potentially tricky situation.
• #20: HCatalog – metadata shared across whole platform File locations become abstract (not hard-coded) Data types become shared (not redefined per tool) Partitioning and HDFS-optimized
• #21: Transition to Andrew
• #22: Last but not least you’ll want to Trace or track the Data Pipeline We trace:
• #29: The first is DR mirroring with Recipes. Actually recipes can be used in number different use cases, but we’ll just focus on mirroring.
• #30: Place holder pic
• #31: Dashboard view Summary counts Inplace filters – by user defined tags
• #32: Entity creation interface is contextual and has field level sematic check to help the user along.
• #33: As you can see on the right – we have the actual XML being generated as the UI field are being filled out.
• #34: This can be help if you want copy portions to skip repeating entity from scratch.
• #36: Lastly the new UI allow to drilll down to the detail level for each entity types.