Introduction to Hadoop
1 like550 views
Eric Mizell Hortonworks - Director of Solution Engineering POSSCON 2015 4/14/2015 Operations/Back-End Track - 2:00 PM Talk
Ad
More Related Content
What's hot (20)
Viewers also liked (17)
Ad
Similar to Introduction to Hadoop (20)
Ad
More from POSSCON (8)
Recently uploaded (20)
Introduction to Hadoop
- 1. Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Introduction to Hadoop Eric Mizell – Director, Solution Engineering Hortonworks. We do Hadoop.
- 2. © Hortonworks Inc. 2012 Page 2
- 3. © Hortonworks Inc. 2012 Page 3
- 4. © Hortonworks Inc. 2012 Page 4
- 5. Page 5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Quick Audience Poll Which best describes how your org is using Hadoop? A. We’re using Hadoop B. We’re in the process of getting Hadoop integrated C. We don’t have Hadoop installed D. What’s Hadoop? E. I don’t know
- 6. Page 6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Big Data, Hadoop, and the Modern Data Architecture
- 7. Page 7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Big Data Explosion Big Data Market Trends & Projections 20% % by which org’s leveraging modern info management systems outperform peers by 2015 !" 1 Zettabyte (ZB) = 1 Billion TBs 15x growth rate of machine generated data by 2020 The US has 1/3 of the world’s data Big Data is 1 of 5 US GDP Game Changers $325 billion incremental annual GDP from big data analytics in retail and manufacturing by 2020
- 8. Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Existing Siloed Data Architectures Under PressureAPPLICATIONS DATA SYSTEM SOURCES Business Analy:cs Custom Applica:ons Packaged Applica:ons Exis:ng Sources (CRM, ERP, Clickstream, Logs) SILO SILO RDBMS SILO SILO SILO SILO EDW MPP Data growth: New Data Types OLTP, ERP, CRM Systems Unstructured docs, emails Clickstream Server logs Social/Web Data Sensor. Machine Data Geoloca:on 85% Source: IDC ?? " Can’t manage new data paradigm " Constrains data to specific schema " Siloed data " Limited scalability " Economically unfeasible " Limited analytics
- 9. Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Hadoop is Driving the New Data-driven Era of IT 1st Era Real-time Data Driven RDBMS 2nd Era 3rd Era Automation + EfficiencyProcessing Power Mainframe GoalDataTechnology
- 10. Page 10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Key Drivers of Hadoop OPERATIONS TOOLS Provision, Manage & Monitor DEV & DATA TOOLS Build & Test DATA SYSTEM REPOSITORIES SOURCES RDBMS EDW MPP APPLICATIONS Business Analy:cs Custom Applica:ons Packaged Applica:ons Unlock New Approach to Analy:cs • Agile analy*cs via “Schema on Read” with ability to store all data in na*ve format • Create new apps from new types of data A Op:mize Investments, Cut Costs • Focus EDW on high value workloads • Use commodity servers & storage to enable all data (original and historical) to be accessible for ongoing explora*on B Enable a Modern Data Architecture • Integrate new & exis*ng data sets • Make all data available for shared access and processing in mul*tenant infrastructure • Batch, interac*ve & real-‐*me use cases • Integrated with exis*ng tools & skills C EXISTING Systems Clickstream Web & Social Geoloca:on Sensor & Machine Server Logs Unstructured YARN: Data Operating System ° ° ° ° ° ° ° ° ° Interactive Real-TimeBatch HDFS: Hadoop Distributed File System
- 11. Page 11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved …to real-time personalizationFrom static branding …to repair before breakFrom break then fix …to designer medicineFrom mass treatment …to automated algorithmsFrom educated investing …to 1x1 targetingFrom mass branding A shift in Advertising A shift in Financial Services A shift in Healthcare A shift in Retail A shift in Manufacturing Hadoop enables organizations to cost effectively store and use all of the data available in a way that shifts the business from… Reactive Proactive Shift to Data-driven Means Treating Data like Capital
- 12. Page 12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Enterprise Goals for the Modern Data Architecture ü Centrally manage new and existing data ü Data needs flexibility and lands in Hadoop without schema ü Prepare data with no predetermined questions ü User self-service – no limit to questions ü Run batch, interactive & real time analytic applications on shared datasets ü Leverage new and existing data center infrastructure investments ü Scalable and affordable; low cost per TB APPLICATIONSDATASYSTEM Business Analytics Custom Applications Packaged Applications RDBMS EDW MPP YARN: Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N Interactive Real-TimeBatch CRM ERP Other 1 ° ° ° ° ° ° ° HDFS (Hadoop Distributed File System) SOURCES EXISTING Systems Clickstream Web & Social Geoloca:on Sensor & Machine Server Logs Unstructured
- 13. Page 13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved YARN and HDP Enables the Modern Data Architecture YARN is the architectural center of Hadoop and HDP • YARN enables a common data set across all applications • Batch, interactive & real-time workloads • Support multi-tenant access & processing HDP enables Apache Hadoop to become Enterprise Viable Data Platform with centralized services • Security • Governance • Operations • Productization Enabled broad ecosystem adoption Hortonworks drove this innovation of Hadoop through YARN Hortonworks Data Platform 2.2 YARN: Data Operating System (Cluster Resource Management) 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° Script Pig SQL Hive Tez Tez Java Scala Cascading Tez ° ° ° ° ° ° ° ° ° ° ° ° ° ° HDFS (Hadoop Distributed File System) Stream Storm Search Solr NoSQL HBase Accumulo Slider Slider SECURITYGOVERNANCE OPERATIONSBATCH, INTERACTIVE & REAL-TIME DATA ACCESS In- Memory Spark Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie Data Workflow, Lifecycle & Governance Falcon Sqoop Flume Kafka NFS WebHDFS Authentication Authorization Audit Data Protection Storage: HDFS Resources: YARN Access: Hive Pipeline: Falcon Cluster: Ranger Cluster: Knox Deployment ChoiceLinux Windows Cloud Others ISV Engines On-Premises
- 14. Page 14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved OPERATIONAL TOOLS DEV & DATA TOOLS INFRASTRUCTURE Modern Data ArchitectureSOURCES EXISTING Systems Clickstream Web &Social Geoloca:on Sensor & Machine Server Logs Unstructured DATASYSTEM RDBMS EDW HANA APPLICATIONS BusinessObjects BI Deep Partnerships Hortonworks engages in deep engineered relationships with the leaders in the data center, such as Microsoft, HP, Teradata, SAS, SAP & Redhat Broad Partnerships Over 600 partners work with us to certify their applications to work with Hadoop so they can extend big data to their users YARN: Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS (Hadoop Distributed File System) Interactive Real-TimeBatch
- 15. Page 15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Hadoop unlocks a new approach: Iterative Analytics ✚ Determine list of ques:ons Design solu:ons Collect structured data Ask ques:ons from list Detect addi:onal ques:ons Current Reality Apply schema on write Dependent on IT Repeatable Process: SQL Only Augment w/ Hadoop Apply schema on read Support range of access patterns to data stored in HDFS: polymorphic access HADOOP Iterate over structure Transform and Analyze batch interactive real-time Right Engine, Right Job in-memory
- 16. Page 16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Hadoop delivers compelling economics ✚ EDW Optimization OPERATIONS 50% ANALYTICS 20% ETL PROCESS 30% OPERATIONS 50% ANALYTICS 50% Current Reality EDW at capacity: some usage from low value workloads Older data archived, unavailable for ongoing exploration Source data often discarded Augment w/ Hadoop Free up EDW resources from low value tasks Keep 100% of source data and historical data for ongoing exploration Mine data for value after loading it because of schema-on-read MPP SAN Engineered System NAS HADOOP Cloud Storage $0 $20,000 $40,000 $60,000 $80,000 $180,000 Fully-loaded Cost Per Raw TB of Data (Min–Max Cost) Commodity Compute & Storage Hadoop Enables Scalable Compute & Storage at a Compelling Cost Structure Hadoop Parse, Cleanse Apply Structure, Transform
- 17. Page 17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved How to Get Started with Hadoop
- 18. Page 18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Try Hadoop Today Download the Hortonworks Sandbox http://hortonworks.com/products/hortonworks-sandbox/ Learn Hadoop Build a Proof of Concept Test New Functionality
- 19. © Hortonworks Inc. 2013 5 Reasons Hadoop is Kicking Cans and Taking Names Hadoop’s momentum is unstoppable as its open source roots grow wildly into enterprises. Its refreshingly unique approach to data management is transforming how companies store, process, analyze, and share big data. Forrester believes that Hadoop will become must-have infrastructure for large enterprises. Here are five reasons firms should adopt Hadoop today: 1. Build a data lake with the Hadoop file system (HDFS) 2. Enjoy cheap, quick processing with MapReduce 3. Data scientists can wrangle big data faster 4. Even the POC can make you money 5. The future of Hadoop is real-time and transactional Page 19 http://blogs.forrester.com/mike_gualtieri/13-10-22-5_reasons_hadoop_is_kicking_can_and_taking_names
- 20. © Hortonworks Inc. 2013 Hadoop Summit 2015 Page 20
- 21. Page 21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved © Hortonworks Inc. 2013 Thank You! Eric Mizell - Director, Solutions Engineering [email protected]