GDPR-focused partner community showcase for Apache Ranger and Apache Atlas

Editor's Notes

• #6: How fast ? 7 months !
• #7: What does ecosystem look like? Connectors for Sqoop, Hive, Storm, Kafka as well as custom integration method to build your own connector via highly scalable REST API. For ex, although there is no first class connector for Spark, you can hook a snippet of code at end of your Spark job to report lineage/metadata info into Atlas. More native connectors being worked for future releases: NiFi and Hbase We also have partner program for ‘Gov ready’ certification and you can see a list of partners who have already built integration Some interesting ones: Talend: data pipelining done in their canvas gets faithfully converted into Atlas lineage graph so we’re able to capture all the steps/transformations/metadata for each of the processes/entities in that chain Dataguise/Waterline do data discovery and are able to publish classification in bulk into Atlas. Same can be done for lineage IGC is special…its joined at the hip with Atlas: they will have one to one model equivalency in terms of backend and will be able to query each other for metadata/lineage etc
• #34: The slide shows the high-level control flow (the title 1st line for each of the 3 boxes): a DMX-h job runs, produces lineage info which is later on published in Atlas More details for each box: DMX-h job executes – currently in the product we are looking at the lineage at the sources/targets level. From the perspective of Atlas, we need to categorize sources/targets that standardized in Atlas, e.g. Hive, HDFS, vs the ones that are not. This is so that DMX-h can later on publish the lineage around these sources/targets as expected by Atlas. DMX-h job produces lineage information – currently this is done for ingestion, and not for distributed executions, and not at field-level. DMX-h job lineage is published into Atlas – DMX-h publishes lineage using (existing) HDFS files and Hive tables entities in Altas, as they are standardized. Other tools (e.g. Hive SQL queries) can use the same HDFS/Hive entities to publish their own lineage, therefore “connecting” to ours from DMX-h We use the REST API to publish the DMX-h lineage. In the product we currently use v1 of the APIs, which is now Legacy, as v2 is the most current. Need to update our product for v2.
• #35: This is a simple DMX-h job that ingests an EBCDIC file into the cluster and converts it to ASCII on the fly.
• #36: Syncsort DMX-h is highly-efficient software with a small footprint, yet it packages the comprehensive support you need to manage, secure and govern your modern data architecture: Manage: Full integration with Apache Ambari Secure: Native LDAP and Kerberos support Integration with Apache Ranger Secure mainframe data access through FTPS and Connect:Direct Govern: Tight integration with HCatalog for metadata management and data lineage Work directly with mainframe data in its native format – preserving data lineage Can tag metadata that contains Personal Identifiable Information which is critical for GDPR compliance (i.e. knowing where personal data is stored)
• #37: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #38: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #39: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #40: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #41: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #42: A better way is needed – so that, just like the chef, we can have a complete view of our data, from the origin to the data hub – and know what has happened to it at every step of the way
• #43: Syncsort/Hortonworks reference architecture Deployed by Ambari On every node Data movement and transformation MapReduce or Spark
• #44: Syncsort/Hortonworks reference architecture Deployed by Ambari On every node Data movement and transformation MapReduce or Spark
• #63: First AI powered all-in-one big data solution Solves the big data myth once and for all Data ingest – Organize – Search – Analyze – Extract All in One Ultra-fast big data visual analytics Unlock the big data complexity Interactive and dynamic user interface fusing Deep Learning with a scalable Data Lake into a ready-to-go Big Data solution
• #64: DSR – Data Subject Rights as it flows through the data assets such as consent
• #65: First AI powered all-in-one big data solution Solves the big data myth once and for all Data ingest – Organize – Search – Analyze – Extract All in One Ultra-fast big data visual analytics Unlock the big data complexity Interactive and dynamic user interface fusing Deep Learning with a scalable Data Lake into a ready-to-go Big Data solution