Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) final
9 likes5,552 views
This document discusses using Hadoop and the Hortonworks Data Platform (HDP) for big data applications. It outlines how HDP can help organizations optimize their existing data warehouse, lower storage costs, unlock new applications from new data sources, and achieve an enterprise data lake architecture. The document also discusses how Talend's data integration platform can be used with HDP to easily develop batch, real-time, and interactive data integration jobs on Hadoop. Case studies show how companies have used Talend and HDP together to modernize their data architecture and product inventory and pricing forecasting.
![Discover.hdp2.2.h base.final[2]](https://cdn.slidesharecdn.com/ss_thumbnails/discover-141218155001-conversion-gate02-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) final (20)
Ad
More from Hortonworks (20)
Recently uploaded (20)
Starting Small and Scaling Big with Hadoop (Talend and Hortonworks webinar)) final
- 1. © Talend 2014 1 Starting Small and Scaling Big with Hadoop November 20, 2014
- 2. © Talend 2014 2 Your Speakers Today Jim Walker Director, Product Marketing Julien Sauvage Director, Product Marketing
- 3. Page 3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved 2013 Digital universe 2.3 Zettabytes 1 Zettabyte (ZB) = 1 million Petabytes (PB); Sources: IDC and IDG Enterprise 85% of growth from new types of data with machine-generated data increasing 15x 2020 Digital universe 40 Zettabytes & Hadoop Market $50B Analysts consensus estimates enterprise data growth of year over year through 2020 50x
- 4. Page 4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved A shift from reactive to proactive interactions HDP and Hadoop allow organizations to shift interactions from… Reactive Post Transaction Proactive Pre Decision …to Real-time Personalization From static branding …to repair before break From break then fix …to Designer Medicine From mass treatment …to Automated Algorithms From Educated Investing …to 1x1 Targeting From mass branding A shift in Advertising A shift in Financial Services A shift in Healthcare A shift in Retail A shift in Telco
- 5. Page 5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved HDP Realized Cost Savings with EDW Optimization Archive Data away from EDW •Move cold or rarely used data to Hadoop as active archive •Store more of data longer Offload costly ETL process •Free your EDW to perform high-value functions like analytics & operations, not ETL. •Use Hadoop for advanced ETL Optimize the value of your EDW •Use Hadoop to refine new data sources, such as web and machine data for new analytical context ANALYTICS DATA SYSTEMS Data Marts Business Analytics Visualization & Dashboards Systems of Record RDBMS ERP CRM Other Clickstream Web & Social Geolocation Sensor & Machine Server Logs Unstructured NEW SOURCES HDP 2.2 ELT ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N Cold Data, Deeper Archive & New Sources Enterprise Data Warehouse Hot Hadoop Helps you optimize and reduce costs associated with your EDW
- 6. Page 6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Realize dramatic savings for cost of storage Cost Efficiencies Reduce costs associated with expensive archive systems •Utilize existing relationships with hardware vendors •Open Source Software Active Archive Provide access to archived data not just collect dust MPP SAN Engineered System NAS HADOOP Cloud Storage $0 $20,000 $40,000 $60,000 $80,000 $180,000 Fully-loaded Cost Per Raw TB of Data (Min–Max Cost) Hadoop Enables Scalable Compute & Storage at a Compelling Cost Structure Storage Costs/Compute Costs from $19/GB to $0.23/GB
- 7. Page 7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Unlock New Applications from New Types of Data INDUSTRY USE CASE Sentiment & Web Clickstream & Behavior Machine & Sensor Geographic Server Logs Structured & Unstructured Financial Services New Account Risk Screens ✔ ✔ Trading Risk ✔ Insurance Underwriting ✔ ✔ ✔ Telecom Call Detail Records (CDR) ✔ ✔ Infrastructure Investment ✔ ✔ Real-time Bandwidth Allocation ✔ ✔ ✔ Retail 360° View of the Customer ✔ ✔ ✔ Localized, Personalized Promotions ✔ Website Optimization ✔ Manufacturing Supply Chain and Logistics ✔ Assembly Line Quality Assurance ✔ Crowd-sourced Quality Assurance ✔ Healthcare Use Genomic Data in Medial Trials ✔ ✔ ✔ Monitor Patient Vitals in Real-Time Pharmaceuticals Recruit and Retain Patients for Drug Trials ✔ ✔ Improve Prescription Adherence ✔ ✔ ✔ ✔ Oil & Gas Unify Exploration & Production Data ✔ ✔ ✔ ✔ Monitor Rig Safety in Real-Time ✔ ✔ ✔ Government ETL Offload/Federal Budgetary Pressures ✔ ✔ Sentiment Analysis for Government Programs ✔
- 8. Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved End Game: Data Lake - An architectural shift SCALE SCOPE Unlocking the Data Lake RDBMS MPP EDW Data Lake Enabled by YARN •Single data repository, shared infrastructure •Multiple biz apps accessing all the data •Enable a shift from reactive to proactive interactions •Gain new insight across the entire enterprise New Analytic Apps or IT Optimization HDP 2.1 Governance & Integration Security Operations Data Access Data Management YARN
- 9. Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Enterprise Goals for the Modern Data Architecture •Consolidate siloed data sets structured and unstructured •Central data set on a single cluster •Multiple workloads across batch interactive and real time •Central services for security, governance and operation •Preserve existing investment in current tools and platforms •Single view of the customer, product, supply chain APPLICATIONS DATA SYSTEM Business Analytics Custom Applications Packaged Applications RDBMS EDW MPP YARN: Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N Interactive Real-Time Batch CRM ERP Other 1 ° ° ° ° ° ° HDFS (Hadoop Distributed File System) SOURCES EXISTING Systems Clickstream Web &Social Geolocation Sensor & Machine Server Logs Unstructured
- 10. Page 10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved HDP delivers a comprehensive data management platform Hortonworks Data Platform 2.2 YARN: Data Operating System (Cluster Resource Management) 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° Script Pig SQL Hive Tez Tez Java Scala Cascading Tez ° ° ° ° ° ° ° ° ° ° ° ° ° ° Others ISV Engines HDFS (Hadoop Distributed File System) Stream Storm Search Solr NoSQL HBase Accumulo Slider Slider SECURITY GOVERNANCE OPERATIONS BATCH, INTERACTIVE & REAL-TIME DATA ACCESS In-Memory Spark Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie Data Workflow, Lifecycle & Governance Falcon Sqoop Flume Kafka NFS WebHDFS Authentication Authorization Accounting Data Protection Storage: HDFS Resources: YARN Access: Hive, … Pipeline: Falcon Cluster: Knox Cluster: Ranger Deployment Choice Linux Windows On-Premises Cloud YARN is the architectural center of HDP Enables batch, interactive and real-time workloads Provides comprehensive enterprise capabilities The widest range of deployment options Delivered Completely in the OPEN
- 11. Page 11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved OPERATIONAL TOOLS DEV & DATA TOOLS INFRASTRUCTURE HDP and Talend in the Modern Data Architecture SOURCES EXISTING Systems Clickstream Web &Social Geolocation Sensor & Machine Server Logs Unstructured DATA SYSTEM RDBMS EDW APPLICATIONS BusinessObjects BI HDP 2.1 Governance & Integration Security Operations Data Access Data Management YARN Hadoop 2.0, YARN Data Quality Pig, Hive, ETL ELT HBase, NoSQL Deep Partnerships Hortonworks engages in deep engineered relationships with the leaders in the data center, such as Microsoft, Teradata, Redhat, HP, SAS & SAP Broad Partnerships Over 600 partners work with us to certify their applications to work with Hadoop so they can extend big data to their users
- 12. © Talend 2014 12 Connecting the Data-Driven Enterprise
- 13. © Talend 2014 13 The Talend Platform
- 14. © Talend 2014 14 Still Hand-Coding Data Integration? Hand-coding Talend Enterprise •Unproductive •Need specialized skills •Hard to maintain •Limited support •800+ drag-n-drop components •Generates optimized code •Collaboration & management •Gold support (SLAs)
- 15. © Talend 2014 15 Encumbered with Legacy ETL? Legacy ETL Talend Enterprise •Proprietary engine •Hard to scale Big Data •Expensive •Open •Generates native code •Low TCO
- 16. © Talend 2014 16 Next big thing SQL ELT DW appliance Future-Proof Architecture ETL Day-to-day integration JAVA Hadoop Highly Scalable MapReduce CAMEL Message transform-ation CAMEL
- 17. © Talend 2014 17 ONE cluster to deploy ONE cluster to manage ONE cluster to monitor ONE cluster to scale ONE cluster to update ONE cluster to pay for! And it will be 100x faster in 2 years Infinite Scale
- 18. © Talend 2014 18 Unlock New Applications from New Types of Data INDUSTRY USE CASE Sentiment & Web Clickstream & Behavior Machine & Sensor Geographic Server Logs Structured & Unstructured Financial Services New Account Risk Screens ✔ ✔ Trading Risk ✔ Insurance Underwriting ✔ ✔ ✔ Telecom Call Detail Records (CDR) ✔ ✔ Infrastructure Investment ✔ ✔ Real-time Bandwidth Allocation ✔ ✔ ✔ Retail 360° View of the Customer ✔ ✔ ✔ Localized, Personalized Promotions ✔ Website Optimization ✔ Manufacturing Supply Chain and Logistics ✔ Assembly Line Quality Assurance ✔ Crowd-sourced Quality Assurance ✔ Healthcare Use Genomic Data in Medial Trials ✔ ✔ ✔ Monitor Patient Vitals in Real-Time Pharmaceuticals Recruit and Retain Patients for Drug Trials ✔ ✔ Improve Prescription Adherence ✔ ✔ ✔ ✔ Oil & Gas Unify Exploration & Production Data ✔ ✔ ✔ ✔ Monitor Rig Safety in Real-Time ✔ ✔ ✔ Government ETL Offload/Federal Budgetary Pressures ✔ ✔ Sentiment Analysis for Government Programs ✔ integration jobs + ++
- 19. © Talend 2014 19 100x performance increase < 1 sec response Address new use cases (last minute defense, dynamic pricing, real-time fraud detection, etc.) Simplify Real-Time Big Data New components for streaming data
- 20. © Talend 2014 20 The Talend Solution Scalable •Generates native code •Future-proof •Built-in data quality •More productive •Open source •Innovative Agile •Open source platform •Learn once •Expand many times Easy •Subscription pricing •Per developer •Predictable cost Lowest TCO The ease of use of the Talend platform allows us to deliver
- 21. © Talend 2014 21 The Three Drivers of Success Product Innovation Market Adoption Industry Recognition Customers Community Partners “Visionary” “Leader” Multi-award winner Big Data Cloud
- 22. Page 22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Customer Case Study Product Inventory and Pricing
- 23. © Talend 2014 23 The Old Way to Do Forecasting Product category “HALLOWEEN”
- 24. © Talend 2014 24 Data Explosion in Size Multiple SKUs Multiple stores Product 2 Product 1 Product 3 Halloween mask Halloween candies Pumpkin 10,000’s 1, 000’s X
- 25. © Talend 2014 25 Need for a Modern Architecture data at rest DAO Cassandra OLTP Hadoop EDW data in motion BI Viz & Analytics Graphical Generates code Runs on Hadoop
- 26. © Talend 2014 26 A New EDW “Eco” System Enterprise Intelligence & Advanced Analytics SSAS Enterprise Data Warehouse Advanced Analytics Platform Data Refinery & Ingest Engine Fast Data Cache
- 27. Page 27 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Talend + Hortonworks = Open = Awesome! •Pure open source governed cluster •Don’t need to recode or reformat data •No vendor lock-in •Subscription models •Most recent releases of Apache projects •We are always aligned and up to date
- 28. Page 28 © Hortonworks Inc. 2011 – 2014. All Rights Reserved The Forrester Wave™ Big Data Hadoop Solutions Q1 2014 “Hortonworks loves and lives open source innovation” World Class Support and Services. Hortonworks' Customer Support received a maximum score and was significantly higher than both Cloudera and MapR A Leader in Hadoop
- 29. © Talend 2014 29 Questions? Jim Walker @jaymce Julien Sauvage @sauvageju
- 30. © Talend 2014 30 Check Out Our Talend + Hortonworks Sandbox! http://www.talend.com