Enabling the Real Time Analytical Enterprise
Download as PPTX, PDF5 likes2,314 views
This document discusses enabling real-time analytics in the enterprise. It begins with an overview of the challenges of real-time analytics due to non-integrated systems, varied data types and volumes, and data management complexity. A case study on real-time quality analytics in automotive is presented, highlighting the need to analyze varied data sources quickly to address issues. The Hortonworks/Attunity solution is then introduced using Attunity Replicate to integrate data from various sources in real-time into Hortonworks Data Platform for analysis. A brief demonstration of data streaming from a database into Kafka and then Hortonworks Data Platform is shown.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Enabling the Real Time Analytical Enterprise (20)
Ad
More from Hortonworks (20)
Recently uploaded (20)
Enabling the Real Time Analytical Enterprise
- 1. Enabling the Real-Time Analytical Enterprise Michael Ger General Manager, Manufacturing & Automotive, Hortonworks Jordan Martz, Director, Technology Solutions Attunity Chris Gambino, Solutions Engineer Hortonworks
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Speakers Michael Ger GM, Industrial Manufacturing and Automotive Solutions Hortonworks Jordan Martz Director of Technology Solutions Attunity Chris Gambino Solutions Engineer Hortonworks
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • The Real-Time Analytics Challenge • Case Study: Real-Time Quality Analytics • The Hortonworks/Attunity Solution • Brief Demonstration • Q & A
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Connected World Connected World Inventory Factories Devices Vehicles 30X Connected Device growth by 2020 (Gartner) 3X growth in connected vehicles by 2022 (PWC) $12M to $209 BILLION growth in RFID by 2021 (McKinsey) 26.9% CAGR growth in Manufacturing IoT thru 2020 (Forbes) Consumers 7X growth in connected population within last 5 years (United Nations)
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Driving Digital, Real-Time Business Processes • Connected Customers, Vehicles, Devices • Socially crowd- sourced requirements • Digital design and analysis • Digital prototypes and tests (simulations) • Connected Factories, Sensors, Devices • Human-robotic interaction • 3D-printing on demand • Connected Trucks, Inventory • Location, traffic, weather-aware distribution • Real-time inventory visibility • Dynamic rerouting • Connected Customers, Devices • Omni- channel demand sensing • Real-Time Recommendations • Connected Assets • Remote service monitoring & delivery • Predictive maintenance • OTA Updates Development Manufacturing Distribution Marketing/Sales Service
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved In Connected World, Time to Insight Is Key Leverage ALL information within your Enterprise? Access the latest, real-time information? Analyze and respond immediately to opportunities and threats? What if You Could…..
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Typical Barriers to Success Non-Integrated Systems Data Variety Data Volume Data Management Complexity and Cost
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved The Opportunity The Real-Time Analytical Organization Instant visibility to trending issues Instant access to data for analysis Collect and Aggregate Data in Real-Time Enable Real-Time Insights “New” data sources (IOT, Social, CX, etc.) Enterprise Transactional Data
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved CASE STUDY: REAL-TIME QUALITY ANALYTICS
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Automotive Quality In The News RecallsCustomer Satisfaction Government Regulation Quality Across Entire Customer Experience Recalls a Growing Occurrence The Watchful Eye
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Early Visibility and Resolution is Key…. Exponential increases in warranty costs by lifecycle phase* • Initial Design $1X • Simulation Testing $10X • Prototype Testing $100X • Initial Production $1,000X The Rule of 10’s *Source: Warranty Week • Warranty Repair $10,000X • Recall $100,000X Early Visibility Minimizes Units in Field!
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What Does it Take to Respond Quickly? What Happened? Why Did It Happen? Contain The Problem
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Installed Base Voice Of Field Voice Of Supplier Production and Operations Voice Of Factory Production and Operations Voice of Vehicle Telematics Voice Of Customer Social + Web (Forums, Surveys, etc.) Call Center Service & Warranty Answers Require Wide Ranging Data • Likes • Dislikes • Sentiment • Complaints • Claims • Man • Method • Material • Machine • Man • Method • Material • Machine • DTC’s • Performance Parameters • VINs • Configurations • Locations Voice Of Field What Happened? Why Did It Happen? Contain The Problem
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved From a Variety of Data Source Types What Happened? Why Did It Happen? Contain The Problem Voice Of Customer Social + Web (Forums, Surveys, etc.) Call Center Service & Warranty Voice of Vehicle Voice Of Factory Voice Of Supplier Production and Operations Installed Base Telematics Production and Operations Voice Of Field SOCIAL, WEB TRANSACTION TRANSACTION TRANSACTION, IOT TRANSACTION IOT TRANSACTION
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • The Real-Time Analytics Challenge • Case Study: Real-Time Quality Analytics • The Hortonworks/Attunity Solution • Brief Demonstration • Q & A
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks Data Platform Reference Architecture - Detailed HCatalog: Shared Table & User Defined Metadata for All Workloads Oozie: Orchestrate ProcessingIngest Security/Governance: Full Stack Authz Policy Definition, Enforcement, Audit Data Cleansing Data Transformation Data Processing Hive LLAP Tooling R ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° HDFS (Hadoop Distributed File System)/ S3 YARN (Cluster Resource Management) Sources Data Science, Machine Learning Data Exploration Model Building Custom Applications Dashboards Analytics, BI, Ad-hoc Exploration Visualization & Reporting Data Exploration SAS Python Cloudbreak: Provision – via Ambari BluePrints and HDP Cluster on AWS, AutoScale nodes in Cluster based on Job Metrics Cubes CDC Vehicle Transactional/ MessagingSystems Iot:Telematics,Social
- 18. © 2016 Attunity Attunity Replicate 2017
- 19. © 2016 Attunity Replicate Product Architecture and Demo
- 20. © 2016 Attunity …Across All Major Platforms SAP Oracle DB2 z/OS DB2 LUW SQL Data Warehouse Exadata Teradata Netezza Vertica Actian Vector Actian Matrix Hortonworks Cloudera MapR Pivotal Hadoop IMS/DB SQL M/P Enscribe RMS VSAM Legacy AWS RDS Salesforce Cloud RDBMS Oracle SQL Server DB2 LUW MySQL PostgreSQL Sybase ASE Informix MariaDB Data Warehouse Exadata Teradata Netezza Vertica Pivotal DB (Greenplum) Pivotal HAWQ Actian Vector Actian Matrix Sybase IQ Hortonworks Cloudera MapR Pivotal Hadoop MongoDB NoSQL AWS RDS/Redshift/EC2 Google Cloud SQL Google Cloud Dataproc Azure SQL Data Warehouse Azure SQL Database Cloud Kafka Message Broker targets Sources RDBMS Oracle SQL Server DB2 iSeries DB2 z/OS DB2 LUW MySQL Sybase ASE Informix
- 21. © 2016 Attunity Feeding the Data Lake with Attunity Replicate Results 4500 applications DB2 MF SQL Oracle • Consolidating massive data volumes for global analytics • Hadoop Data Lake with Kafka • Minimizing labor and cost • Realizing faster insights and competitive advantage Fortune 100 auto maker
- 22. © 2016 Attunity Feeding the Data Lake with Attunity Replicate Heterogeneous applications DB2 MF SQL Oracle •Combining massive data volumes for global analytics • SAP HANA, Microsoft Azure •Standardize ingestion across the data lake •Initial use cases include – warranty analysis, remanufacturing analysis Major Construction Equipment Manufacturer DB2 LUW IMS VSAM Teradata
- 23. © 2016 Attunity Data Lake Ingests with Attunity Replicate: On-Prem & Clouds Transfer TransformFilter Batch CDC Incremental In-Memory File Channel Batch Hadoop Files RDBMS Data Warehouse Mainframe Cloud On-prem Cloud On-prem Hadoop Files RDBMS Data Warehouse Kafka Persistent Store
- 24. © 2016 Attunity Attunity Solutions for SAP Hadoop & Big Data Data Warehousing Database Cloud SAP 1. SAP Test Data Management 2. SAP HANA Data Integration 3. Enabling SAP Analytics
- 25. © 2016 Attunity Replicate for SAP TransformFilter Batch CDC Incremental In-Memory File Channel Batch Attunity Replicate for SAP – Data Flow Persistent Store Extract relationships for Pool and Cluster Tables Navigate and select SAP business objects Automated ABAP Mapping and Change- Data-Capture for Pool and Cluster tables 1. Replicate for SAP UI 2. Replicate for SAP RFC Calls RDBMS (Oracle, DB2, etc.) Redo/ Archive logs or Journal File ---------------- Transparent Tables On Premises Hadoop RDBMS Data WarehouseKafka Cloud
- 26. © 2016 Attunity Attunity Replicate Go agile with modern and automated integration • No manual coding or scripting • Automated end-to-end • Optimized and configurable Hadoop Files RDBMS EDW Mainframe • Target schema creation • Heterogeneous data type mapping • Batch to CDC transition • DDL change propagation • Filtering • Transformations Hadoop Files RDBMS EDW Kafka
- 27. © 2016 Attunity Zero-footprint Architecture • Lower impact on IT • No software agents on sources and targets for mainstream databases • Replicate data from 100’s of source systems with easy configuration • No software upgrades required at each database source or target Hadoop Files RDBMS EDW Mainframe • Log based • Source specific optimization Hadoop Files RDBMS EDW Kafka
- 28. © 2016 Attunity Hortonworks HDF + Attunity Replicate
- 29. © 2016 Attunity The Connected Data Platform
- 30. © 2016 Attunity The Connected Data Architecture & Attunity SOURCES OLTP, ERP, CRM Systems Documents, Emails Web Logs, Click Streams Social Networks Machine Generated Sensor Data Geolocation Data Data Integration & Ingests Attunity Replicate for HDP and HDF Accelerate time-to-insights by delivering solutions faster, with fresher data, from many sources - Automated data ingest - Incremental data ingest (CDC) - Broad support for many sources
- 31. © 2016 Attunity In Memory and File Optimized Data Transport Enterprise-class CDC for Data At Rest and Data In Motion R1 R1 R2 R1 R2 R1 R2Batch CDC Data Warehouse Ingest-Merge SQL n 2 1 SQL SQL Transactional CDC Message Encoded CDC Data Sources Attunity Replicate – Change Processing CDC Many Databases and Data Warehouses ....
- 32. © 2016 Attunity Demand • Easy ingest + CDC • Real-time processing • Real-time monitoring • Real-time Hadoop • Scalable to 1000’s applications • One publisher – multiple Consumers Attunity - Replicate • Direct integration using Kafka APIs • In-memory optimized data streaming • Support for multi-topic and multi- partitioned data publication • Full load and CDC • Integrated management and monitoring via GUI Kafka and Real-time Streaming
- 33. © 2016 Attunity T1/P0 T2/P1 T3/P0 Broker 1 Attunity Replicate for Kafka - Architecture M0 M1 M2 M3 M4 M5 M6 M7 M8 M0 M1 M2 M3 M4 M5 M0 M1 M2 M3 M4 M5 M6 M7 T1/P1 T2/P0 Broker 2 M0 M1 M2 M3 M4 M0 M1 M2 M3 M4 M5 M6
- 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda • The Real-Time Analytics Challenge • Case Study: Real-Time Quality Analytics • The Hortonworks/Attunity Solution • Brief Demonstration • Q & A
- 35. © 2016 Attunity CDC Demo: Data Streaming into Kafka HDF HDP MSG n 2 1 MSG MSG Data Streaming Transaction logs In memory optimised metadata management and data transport Bulk Load MSG n 2 1 MSG MSG Data Streaming Message broker Message broker Example DB:
- 36. © 2016 Attunity Demo: Data Streaming into Kafka HDF HDP
- 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
- 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You! Q/A Hortonworks.com Attunity.com
Editor's Notes
- #18: Nifi becomes the central ingest mechanism for all data coming into the environment. Nifi can be instantiated as its own WebServer listening to PUT/POST requests from different sources, Nifi can pull from different sources such as S3, SFTP, HTTPS, RDBMS, and more. Once the data is inside of Nifi we can do simple event processing, launch parsers that PerkinElmer has already created, perform data enrichment before landing into a Kafka queue. NOTE: throughout the duration of data lifecycle within Nifi and within HDP we will be tracking lineage such as where the data came from and how it was manipulated. Placing the data in a Kafka queue will allow other engines to easily pick up the data to begin working with it such as Spark. Spark will be used as the heavy lifting ETL processing along with SQL, and Machine Learning it provides. Leveraging Spark on Hadoop allows Spark to adhere to Security (file,folder, column level security). Writing the data out as ORC file formats keeps the data more flexible to be used with other options such as Hive. When the data is processed through Spark and stored down through the HiveContext to ORC file formats (or just left in spark) we can then expose to external sources such as SpotFire for analysts to do analysis on.
- #24: What the industry cares about: Hadoop has moved out of test Enterprise Use Case Closer to Production Business impact Enterprise + Real-time VS Sqoop + Batch Attunity + Replicate High performance connectivity to Hadoop though native APIs for data ingest and publication Automated schema generation in Hcatalog Drag & drop configuration with Click-2-Replicate design High-speed data load options: Full reload with overwrite Insert only appends Change Data Capture(CDC) In-memory data filtering and transformation Monitoring dashboard with web-based metrics, alerts and log file management
- #28: Another advantage of Replicate is the agentless data replication for mainstream database systems. Recently a customer needed to ingest data from 4500 applications across hundreds of databases into Hadoop. With Replicate they are able to do this without installing an agent on each source system because Replicate extracts source logs remotely in an optimized manner and processes the data in-memory on the Replicate server. This also means that maintaining the product is simplified since it does not requiring maintaining and upgrading software agents each source or target system.
- #29: ATTU/CDC Automated data ingest Incremental updates with Change Data Capture (CDC) Broad support for many enterprise data sources HDP/HDF Rapid deployments of HUGE data lakes Continuous data refresh for RELEVANT analytics COMPLETE datasets across databases, DWs and mainframes
- #30: HDF and HDP form the Connected Data Platform Data in Motion (connected, real-time, tracked) and Data at Rest (massive scale analysis, retention, security) Modern Data Applications are built on the Connected Data Platform Metron for example Customer built applications
- #32: One of the reasons several large technology companies trust and rely on Attunity for their own solutions is because of the robust CDC capability that Replicate provides. There are several options that are built into the product that provide flexible and optimized ways to implement change data capture. In addition to applying transactions in real-time and in-order, Replicate can handle varying volumes of changes on the source systems by applying the changes in optimized batches to improve throughput and latency In order to provide high-speed data loads into data warehouse appliances, Replicate is integrated with native data warehouse loaders for fast data ingestion into the target and then changes are merged in the target. It does not rely on sub optimal ODBC for data loading into the ware house systems. And recently, Attunity added support to write changes in message encoded format that can be published to Kafka message brokers as well.
- #36: Data transport and integration Log data Database changes Sensors and device data Monitoring streams Call data records Stock ticker data Real-time stream processing Monitoring Asynchronous applications Fraud and security
- #37: Data transport and integration Log data Database changes Sensors and device data Monitoring streams Call data records Stock ticker data Real-time stream processing Monitoring Asynchronous applications Fraud and security