A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks and Red Hat
5 likes2,521 views
The document outlines the collaborative efforts of Hortonworks, Cisco, and Red Hat to provide an open-source big data solution utilizing Hadoop for scalability and efficiency. It highlights the challenges with traditional systems in managing data and introduces a modern architecture that centralizes data access and analytics. The presentation emphasizes the benefits of the Hortonworks Data Platform (HDP), which is designed for versatile data handling and cost optimization in enterprise environments.
A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks and Red Hat
- 1. Page 1 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved A Comprehensive Approach to Building Your Big Data Solution We do Hadoop.
- 2. Page 2 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Speakers ย โฏ Hortonworks โฆโฏ Ali Bajwa, Senior Partner Solution Engineer ย โฏ Red Hat โฆโฏ Irshad Raihan, Senior Principal, Product Marketing ย โฏ Cisco โฆโฏ Ron Graham, Big Data Analytics Engineer
- 3. Page 3 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Partnership 100% ย open ย source ย Hadoop ย Distribu5on, ย ย Support ย and ย Training ย ย Middleware, ย Storage, ย PaaS, ย IaaS ย UCS ย Integrated ย Infrastructure ย For ย Big ย Data ย CISCO, ย HORTONWORKS ย AND ย RED ย HAT ย ARE ย PARTNERING ย TO ย HELP ย YOU ย BUILD ย YOUR ย BIG ย DATA ย SOLUTION ย AND ย REACH ย MASSIVE ย SCALABILITY, ย SUPERIOR ย EFFICIENCY ย AND ย DRAMATICALLY ย LOWER ย TOTAL ย COST ย OF ย OWNERSHIP ย THANKS ย TO ย A ย VALIDATED ย JOINT ย ARCHITECTURE.
- 4. Page 4 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Traditional systems under pressure Challenges โขโฏ Constrains data to app โขโฏ Canโt manage new data โขโฏ Costly to Scale Business Value Clickstream Geolocation Web Data Internet of Things Docs, emails Server logs 2012 2.8 Zettabytes 2020 40 Zettabytes LAGGARDS INDUSTRY LEADERS 1 2 New Data ERP CRM SCM New Traditional
- 5. Page 5 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Modern Data Architecture emerges to unify data & processing Modern Data Architecture โขโฏ Enable applications to have access to all your enterprise data through an efficient centralized platform โขโฏ Supported with a centralized approach governance, security and operations โขโฏ Versatile to handle any applications and datasets no matter the size or type Clickstream ย Web ย ย & ย Social ย Geoloca3on ย Sensor ย ย & ย Machine ย Server ย ย Logs ย Unstructured ย SOURCES Existing Systems ERP ย CRM ย SCM ย ANALYTICS Data Marts Business Analytics Visualization & Dashboards ANALYTICS Applications Business Analytics Visualization & Dashboards ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ HDFS (Hadoop Distributed File System) YARN: Data Operating System Interactive Real-TimeBatch Partner ISVBatch BatchMP P ย EDW ย
- 6. Page 6 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Hadoop Driver: Cost optimization Archive Data off EDW Move rarely used data to Hadoop as active archive, store more data longer Offload costly ETL process Free your EDW to perform high-value functions like analytics & operations, not ETL Enrich the value of your EDW Use Hadoop to refine new data sources, such as web and machine data for new analytical context ANALYTICS Data Marts Business Analytics Visualization & Dashboards HDP helps you reduce costs and optimize the value associated with your EDW ANALYTICSDATASYSTEMS Data Marts Business Analytics Visualization & Dashboards HDP 2.2 ELT ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ N Cold Data, Deeper Archive & New Sources Enterprise Data Warehouse Hot MPP In-Memory Clickstream ย Web ย ย & ย Social ย Geoloca3on ย Sensor ย ย & ย Machine ย Server ย ย Logs ย Unstructured ย Existing Systems ERP ย CRM ย SCM ย SOURCES
- 7. Page 7 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Hadoop Driver: Enabling the data lakeSCALE SCOPE Data Lake Definition โขโฏ Centralized Architecture Multiple applications on a shared data set with consistent levels of service โขโฏ Any App, Any Data Multiple applications accessing all data affording new insights and opportunities. โขโฏ Unlocks โSystems of Insightโ Advanced algorithms and applications used to derive new value and optimize existing value. Drivers: 1.โฏ Cost Optimization 2.โฏ Advanced Analytic Apps Goal: โขโฏ Centralized Architecture โขโฏ Data-driven Business DATA LAKE Journey to the Data Lake with Hadoop Systems of Insight
- 8. Page 8 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Only HDP delivers a Centralized Architecture HDP is uniquely built around YARN serving as a data operating system that provides multi-tenant Resource Management, consistent Governance & Security and efficient Operations services across Hadoop applications. Hortonworks Data Platform YARN Data Operating System โขโฏ A centralized architecture of consistent enterprise services for resource management, security, operations, and governance. โขโฏ The versatility to support multiple applications and diverse workloads from batch to interactive to real- time, open source and commercial. Key Benefits โขโฏ Multiple applications on a shared data set with consistent levels of service: a multitenant data platform. โขโฏ Provides a shared platform to enable new analytic applications. โขโฏ Delivers maximum cost efficiency for cluster resource management. Fewer servers fewer nodes. Storage YARN: Data Operating System Governance Security Operations Resource Management Existing Applications New Analytics Partner Applications Data Access: Batch, Interactive & Real-time
- 9. Page 9 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved HDP delivers a completely open data platform Hortonworks Data Platform 2.2 Hortonworks Data Platform provides Hadoop for the Enterprise: a centralized architecture of core enterprise services, for any application and any data. Completely Open โขโฏ HDP incorporates every element required of an enterprise data platform: data storage, data access, governance, security, operations โขโฏ All components are developed in open source and then rigorously tested, certified, and delivered as an integrated open source platform thatโs easy to consume and use by the enterprise and ecosystem. YARN: Data Operating System (Cluster Resource Management) 1 ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ApachePig ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ ยฐ HDFS (Hadoop Distributed File System) GOVERNANCE BATCH, INTERACTIVE & REAL-TIME DATA ACCESS Apache Falcon ApacheHive Cascading ApacheHBase ApacheAccumulo ApacheSolr ApacheSpark ApacheStorm Apache Sqoop Apache Flume Apache Kafka SECURITY Apache Ranger Apache Knox Apache Falcon OPERATIONS Apache Ambari Apache Zookeeper Apache Oozie
- 10. Page 10 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved HDP: Any Data, Any Application, Anywhere Any Application โขโฏ Deep integration with ecosystem partners to extend existing investments and skills โขโฏ Broadest set of applications through the stable of YARN-Ready applications Any Data Deploy applications fueled by clickstream, sensor, social, mobile, geo-location, server log, and other new paradigm datasets with existing legacy datasets. Anywhere Implement HDP naturally across the complete range of deployment options Clickstream ย Web ย ย & ย Social ย Geoloca3on ย Internet ย of ย Things ย Server ย ย Logs ย Files, ย emails ย ERP ย CRM ย SCM ย hybrid commodity appliance cloud Over 70 Hortonworks Certified YARN Apps
- 11. Page 11 ยฉ Hortonworks Inc. 2011 โ 2014. All Rights Reserved Open Source IS the standard for platform technology Modern platform standards are defined by open communities For Hadoop, the ASF provides guidelines and a governance framework and the open community defines the standards for Hadoop. Roadmap matches user requirements not vendor monetization requirements Hortonworks Open Source Development Model yields unmatched efficiency โขโฏ Infinite number of developers under governance of ASF applied to problem โขโฏ End users motivated to contribute to Apache Hadoop as they are consumers โขโฏ IT vendors motivated to align with Apache Hadoop to capture adjacent opportunities Hortonworks Open Source Business Model de-risks investments โขโฏ Buying behavior changed: enterprise wants support subscription license โขโฏ Vendor needs to earn your business, every year is an election year โขโฏ Equitable balance of power between vendor and consumer โขโฏ IT vendors want platform technologies to be open source to avoid lock-in
- 12. TITLE SLIDE: HEADLINE Presenter name Title, Red Hat Date Red ย Hat ย Big ย Data ย Open ย the ย possibili5es ย of ย your ย data ย
- 13. 13 Big ย Data ย innova3on ย cannot ย happen ย in ย a ย bubble ย Strong ย partnerships ย with ย industry ย leaders ย and ย open ย source ย communi5es ย
- 14. 14 Business ย User ย Architect ย Data ย Center ย Operator ย App ย Developer ย Mul5ple ย Silos. ย Mul5ple ย Views. ย Mul5ple ย Goals. ย The ย Old ย Data ย Lifecycle ย Manage ย ย Build ย ย Code ย Query ย
- 15. 15 Business ย User ย Architect ย Data ย Center ย Operator ย App ย Developer ย One ย Language. ย One ย View. ย One ย Goal. ย The ย New ย Data ย Lifecycle ย Ingest ย Integrate ย Act ย Discover ย
- 16. 16 Lack ย of ย agile, ย open, ย and ย cost ย e๏ฌec5ve ย enterprise-ยญโgrade ย solu5ons ย Barriers ย to ย Big ย Data ย Success ย I ย want ย more ย than ย canned ย BI ย queries ย I ย am ย locked ย into ย a ย vendor ย stack ย I ย want ย to ย use ย my ย favorite ย dev ย framework ย I ย need ย to ย integrate ย data ย across ย silos ย Business ย User ย Architect ย Data ย Center ย Operator ย App ย Developer ย
- 17. 17 Business ย User ย Architect ย Data ย Center ย Operator ย App ย Developer ย Ingest ย Integrate ย Act ย Discover ย Big ย Data ย Solu3ons ย from ย Red ย Hat ย
- 18. Integrated ย Big ย Data ย PlaOorm ย ย Cisco UCS Integrated Infrastructure for Big Data Hadoop Compatible File System Red Hat Storage Hadoop Data Processing Map/Reduce YARN Analytics Operating System Red Hat Enterprise Linux Cloud Red Hat Enterprise Linux OpenStack Platform Operating EnvironmentData Integration & Application Development Application Platform- as-a-Service OpenShift by Red Hat Data Integration and Data Services Red Hat JBoss Data Virtualization Data Caching Red Hat JBoss Data Grid Business Rules Mgmt Red Hat JBoss BRMS Development Red Hat JBoss Developer Studio Hadoop Distributed File System Management HortonworksCisco Red Hat Data Integration and Data Services Composite Cloud Cisco OpenStack Pig Spark Storm HBase Tez Hive Cisco Security Suite CiscoUCSDirectoryExpress CiscoUnifiedManagement Ambari Virtualization Red Hat Enterprise Virtualization
- 19. Software and Solutions Innovation Empowering Whatโs Next Ron Graham Big Data Analytics Engineer Hardware Architecture Cisco UCS with Big Data
- 20. 20ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next Why Cisco UCS for Big Data? โขโฏ Manageability โขโฏ Save time with UCS Manager โขโฏ Enables consistent and rapid deployments using UCS Service profiles โขโฏ Offers operational simplification โขโฏ Delivers a modular solution โขโฏ Scalability โขโฏ Performance SIM Card Identity for a phone Service Profile Identity for a server
- 21. 21ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next โขโฏ End to end provisioning, installation, and monitoring tool for Hadoop Clusters โขโฏ Better business outcomes with faster time to value from Big Data โขโฏ Provides appliance like experience with out inflexibilities โขโฏ Centralized visibility across Hadoop and physical infrastructure โขโฏ Powerful interface for further integration into third party tools and services UCS Director Express for Big Data End to end solution for Hadoop
- 22. 22ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next Powering Big Data and Analytics UCS ย B200 ย Scale-ยญโout ย Analy5cs ย Big ย Data ย with ย EMC ย Isilon ย and ย VCE ย Invicta ย (Fast ย Data) ย UCS ย C240 ย (Hadoop, ย NoSQL ย MPP) ย UCS ย Manager, ย Director, ย Express, ย Central, ย Redhat ย ย ACl ย C/B460 ย (In-ยญโ memory ย Analy5cs) ย UCS ย C3160, ย C3260 ย (Hadoop) ย UCS ย C220 ย (real-ยญโ5me, ย streaming) ย FlexPod ย Select ย with ย NetApp ย E-ยญโSeries ย UCS ย Mini ย (All-ยญโin-ยญโone ย at ย Edge) ย UCS ย M-ยญโSeries ย (Massive ย scale-ยญโout) ย Ac5an, ย DataStax, ย Hortonworks, ย MongoDB, ย Pivotal,SAP, ย SAS, ย Splunk ย ย Cisco, ย Elas5c ย Search, ย IBM, ย Informa5ca, ย MicrosoZ, ย MicroStrategy ย , ย Oracle, ย SAP, ย SAS ย ย and ย others ย Complete ย and ย Industry ย leading ย Por[olio ย Ecosystem ย Partners ย ISV ย Partners ย Infrastructure ย Management ย Data ย Management ย Applica5ons ย
- 23. 23ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next DESIGNS Big Data Cisco Validated Designs for leading big data platforms can be found at: www.cisco.com/go/bigdata Cisco Validated Designs Accelerate Deployment
- 24. 24ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next Server 8x UCS C220 M4 CPU 2 x Intel Xeon E5-2620 v3 (15M Cache, 2.40 GHz) Memory 256GB Storage 8 1.2-TB 10K SAS SFF HDD Starter High Performance Server 8x UCS C220 M4 CPU 2 x Intel Xeon E5-2680 v3 (30M Cache, 2.50 GHz) Memory 384GB Storage 2 1.2-TB 10K SAS SFF HDD, 6 400- GB SAS SSD Performance Optimized Capacity Optimized Extreme Capacity Server 16x UCS C240 M4 CPU 2 x Intel Xeon E5-2680 v3 (30M Cache, 2.50 GHz) Memory 256GB Storage 2 120-GB SATA SSD, 24 1.2-TB 10K SAS SFF HDD Server 16x UCS C240 M4 CPU 2 x Intel Xeon E5-2620 v3 (15M Cache, 2.40 GHz) Memory 128GB Storage 2 120-GB SATA SSD. 12 4-TB 7.2K SAS SFF HDD Server 2x UCS C3160 CPU 2 x Intel Xeon E5-2695 v2 (30M Cache, 2.40 GHz) Memory 256GB Storage 2 120-GB SATA SSD, 60 4-TB 7.2K SAS SFF HDD Cisco UCS CPA for Big Data v3 Reference Architecture and Bundles
- 25. 25ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next 2x UCS 6296 Series Fabric Interconnect UCS Manager โขโฏ UCS Domain (68 Servers) โขโฏ Manage by UCS Manager โขโฏ 2.8 PB of storage โขโฏ HDP 2.2 โขโฏ Tiered Storage โขโฏ Tez โขโฏ RHEL 6.5 โขโฏ Dual 10G Network โขโฏ 17 Servers Per Rack UCS C240 M4 2x E5-2680 v3 256GB Memory Cisco 12Gb/s SAS Raid Controller 2x 120GB STAT SSD 24x 1.2TB 10k SAS 2x Cisco UCS VIC 1227 UCS C3160 2x E5-2695 v2 256GB Memory Cisco 12Gb/s SAS Raid Controller 2x 120GB SATA SSD 60x 4TB 7.2k SAS SFF 2x Cisco UCS VIC 1227 / 17 10Gb Ethernet / 17 10Gb Ethernet 64 Node Cluster Configuration
- 26. 26ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next UCSD Express UCS 6200 Series Fabric Interconnect UCS Manager UCS C240 M4 Series Rack Server UCS C3160 Rack Server Apache Ambari Unified Management Programmability, Scalability and Automation
- 27. 27ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next UCS 6200 Series Fabric Interconnect UCS C240 M4 Series Rack Server UCS C3160 Rack Server Data Data Data Cold n replicas on Archive Warm 1 replicas on Disk, n-1 on Archive Hot All (n) replicas on Disk Cold Hot Policy Hot - for both storage and compute. The data that is popular and still being used for processing will stay in this policy. When a block is hot, all replicas are stored in DISK. Warm - partially hot and partially cold. When a block is warm, some of its replicas are stored in DISK and the remaining replicas are stored in ARCHIVE. Cold - only for storage with limited compute. The data that is no longer being used, or data that needs to be archived is moved from hot storage to cold storage. When a block is cold, all replicas are stored in ARCHIVE. Multi-tiered Storage Architecture Multi-temperature Policy
- 28. 28ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Software and Solutions Innovation Empowering Whatโs Next UCS 6200 Series Fabric Interconnect UCS C240 M4 Series Rack Server UCS C3160 Rack Server Data Data Data Cold n replicas on Archive Warm 1 replicas on Disk, n-1 on Archive Hot All (n) replicas on Disk Cold Hot Mover โ A new data migration tool It periodically scans the files in HDFS to check if the block placement satisfies the storage policy. For the blocks violating the storage policy, it moves the replicas to a different storage type in order to fulfill the storage policy requirement. A C D A C D E A C D E N N N N E Multi-tiered Storage Architecture Multi-temperature Policy
- 29. Page 29 ยฉ Hortonworks Inc. 2011 โ 2015. All Rights Reserved Next Stepsโฆ Download the Hortonworks Sandbox Learn Hadoop Build Your Analytic App Try Hadoop Learn more with our partnerships http://hortonworks.com/partner/cisco/ http://hortonworks.com/partner/redhat/ Joint CVD bit.ly/Cisco-CVD
- 30. 30ยฉ 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential โขโฏ Cisco Live! in San Diego โ June 7 - 11 โขโฏ Hadoop Summit in San Jose โ June 9 โ 11 โขโฏ Red Hat Summit in Boston - June 23-26 More information about Red Hatโs Big Data solutions please visit: โขโฏ redhat.com/bigdata โขโฏ redhatstorage.redhat.com/category/big-data โขโฏ redhat.com/en/insights/big-data More information about Ciscoโs Big Data and Analytics Offers please visit: โขโฏ www.cisco.com/go/bigdata and www.cisco.com/go/bigdata_design โขโฏ http://blogs.cisco.com/author/raghunathnambiar โขโฏ bit.ly/Cisco-CVD 30 Meet us in person!