A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks and Red Hat

© Hortonworks Inc. 2011 – 2014. All Rights Reserved
A Comprehensive Approach to Building
Your Big Data Solution
We do Hadoop.

Speakers

  Hortonworks
◦  Ali Bajwa, Senior Partner Solution Engineer

  Red Hat
◦  Irshad Raihan, Senior Principal, Product Marketing

  Cisco
◦  Ron Graham, Big Data Analytics Engineer

Partnership
100%
open
source
Hadoop
Distribu5on,

Support
and
Training

Middleware,
Storage,
PaaS,
IaaS

UCS
Integrated
Infrastructure

For
Big
Data

CISCO,
HORTONWORKS
AND
RED
HAT
ARE
PARTNERING
TO
HELP
YOU

BUILD
YOUR
BIG
DATA
SOLUTION
AND
REACH
MASSIVE
SCALABILITY,

SUPERIOR
EFFICIENCY
AND
DRAMATICALLY
LOWER
TOTAL
COST
OF

OWNERSHIP
THANKS
TO
A
VALIDATED
JOINT
ARCHITECTURE.

Traditional systems under pressure
Challenges
•  Constrains data to app
•  Can’t manage new data
•  Costly to Scale
Business Value
Clickstream
Geolocation
Web Data
Internet of Things
Docs, emails
Server logs
2012
2.8 Zettabytes
2020
40 Zettabytes
LAGGARDS
INDUSTRY
LEADERS
1
2 New Data
ERP CRM SCM
New
Traditional

Modern Data Architecture emerges to unify data & processing
Modern Data Architecture
•  Enable applications to have access to
all your enterprise data through an
efficient centralized platform
•  Supported with a centralized
approach governance, security and
operations
•  Versatile to handle any applications
and datasets no matter the size or
type
Clickstream
Web

&
Social

Geoloca3on
Sensor

&
Machine

Server

Logs

Unstructured

SOURCES
Existing Systems
ERP
CRM
SCM

ANALYTICS
Data
Marts
Business
Analytics
Visualization
& Dashboards
ANALYTICS
Applications
Business
Analytics
Visualization
& Dashboards
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
HDFS
(Hadoop Distributed File System)
YARN: Data Operating System
Interactive Real-TimeBatch Partner ISVBatch BatchMP
P

EDW

Hadoop Driver: Cost optimization
Archive Data off EDW
Move rarely used data to Hadoop as active
archive, store more data longer
Offload costly ETL process
Free your EDW to perform high-value functions
like analytics & operations, not ETL
Enrich the value of your EDW
Use Hadoop to refine new data sources, such as
web and machine data for new analytical context
ANALYTICS
Data
Marts
Business
Analytics
Visualization
& Dashboards
HDP helps you reduce costs and optimize the value associated with your EDW
ANALYTICSDATASYSTEMS
Data
Marts
Business
Analytics
Visualization
& Dashboards
HDP 2.2
ELT
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
N
Cold Data,
Deeper Archive
& New Sources
Enterprise
Data
Warehouse
Hot
MPP
In-Memory
Clickstream
Web

&
Social

Geoloca3on
Sensor

&
Machine

Server

Logs

Unstructured

Existing Systems
ERP
CRM
SCM

SOURCES

Hadoop Driver: Enabling the data lakeSCALE
SCOPE
Data Lake Definition
•  Centralized Architecture
Multiple applications on a shared data set
with consistent levels of service
•  Any App, Any Data
Multiple applications accessing all data
affording new insights and opportunities.
•  Unlocks ‘Systems of Insight’
Advanced algorithms and applications
used to derive new value and optimize
existing value.
Drivers:
1.  Cost Optimization
2.  Advanced Analytic Apps
Goal:
•  Centralized Architecture
•  Data-driven Business
DATA
LAKE
Journey to the Data Lake with Hadoop
Systems of Insight

Only HDP delivers a Centralized Architecture
HDP is uniquely built around YARN serving as a data operating system that provides multi-tenant Resource
Management, consistent Governance & Security and efficient Operations services across Hadoop applications.
Hortonworks Data Platform
YARN
Data Operating System
•  A centralized architecture of
consistent enterprise
services for resource
management, security,
operations, and
governance.
•  The versatility to support
multiple applications and
diverse workloads from
batch to interactive to real-
time, open source and
commercial.
Key Benefits
•  Multiple applications on a
shared data set with consistent
levels of service: a multitenant
data platform.
•  Provides a shared platform to
enable new analytic
applications.
•  Delivers maximum cost
efficiency for cluster resource
management. Fewer servers
fewer nodes.
Storage
Governance Security
Operations
Resource Management
Existing
Applications
New
Analytics
Partner
Applications
Data Access: Batch, Interactive & Real-time

HDP delivers a completely open data platform
Hortonworks Data Platform 2.2
Hortonworks Data Platform provides Hadoop for the Enterprise: a centralized architecture
of core enterprise services, for any application and any data.
Completely Open
•  HDP incorporates every element
required of an enterprise data
platform: data storage, data access,
governance, security, operations
•  All components are developed in
open source and then rigorously
tested, certified, and delivered as
an integrated open source platform
that’s easy to consume and use by
the enterprise and ecosystem.
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
ApachePig
° °
° °
° ° °
° ° °
HDFS
(Hadoop Distributed File System)
GOVERNANCE BATCH, INTERACTIVE & REAL-TIME DATA ACCESS
Apache Falcon
ApacheHive
Cascading
ApacheHBase
ApacheAccumulo
ApacheSolr
ApacheSpark
ApacheStorm
Apache Sqoop
Apache Flume
Apache Kafka
SECURITY
Apache Ranger
Apache Knox
Apache Falcon
OPERATIONS
Apache Ambari
Apache
Zookeeper
Apache Oozie

HDP: Any Data, Any Application, Anywhere
Any Application
•  Deep integration with ecosystem
partners to extend existing
investments and skills
•  Broadest set of applications through
the stable of YARN-Ready applications
Any Data
Deploy applications fueled by clickstream, sensor,
social, mobile, geo-location, server log, and other
new paradigm datasets with existing legacy
datasets.
Anywhere
Implement HDP naturally across the
complete range of deployment options
Clickstream
Web

&
Social

Geoloca3on
Internet
of

Things

Server

Logs

Files,
emails
ERP
CRM
SCM

hybrid
commodity appliance cloud
Over 70 Hortonworks Certified YARN Apps

Open Source IS the standard for platform technology
Modern platform standards are defined by open communities
For Hadoop, the ASF provides guidelines and
a governance framework and the open
community defines the standards for Hadoop.
Roadmap matches user
requirements not vendor
monetization requirements
Hortonworks Open Source Development Model yields unmatched
efficiency
•  Infinite number of developers under governance of ASF applied to problem
•  End users motivated to contribute to Apache Hadoop as they are consumers
•  IT vendors motivated to align with Apache Hadoop to capture adjacent opportunities
Hortonworks Open Source Business Model de-risks investments
•  Buying behavior changed: enterprise wants support subscription license
•  Vendor needs to earn your business, every year is an election year
•  Equitable balance of power between vendor and consumer
•  IT vendors want platform technologies to be open source to avoid lock-in

TITLE SLIDE: HEADLINE
Presenter name
Title, Red Hat
Date
Red
Hat
Big
Data

Open
the
possibili5es
of
your
data

13
Big
Data
innova3on
cannot
happen
in
a
bubble

Strong
partnerships
with
industry
leaders
and
open
source
communi5es

14
Business
User
Architect
Data
Center
Operator
App
Developer

Mul5ple
Silos.
Mul5ple
Views.
Mul5ple
Goals.

The
Old
Data
Lifecycle

Manage

Build

Code
Query

15
Business
User

Architect

Data
Center
Operator

App
Developer

One
Language.
One
View.
One
Goal.

The
New
Data
Lifecycle

Ingest
Integrate

Act
Discover

16
Lack
of
agile,
open,
and
cost
eﬀec5ve
enterprise-‐grade
solu5ons

Barriers
to
Big
Data
Success

I
want
more
than

canned
BI
queries

I
am
locked
into
a

vendor
stack

I
want
to
use
my
favorite

dev
framework

I
need
to
integrate

data
across
silos

Business
User

Architect

Data
Center
Operator

App
Developer

17
Business
User

Architect

Data
Center
Operator

App
Developer

Ingest

Integrate

Act

Discover

Big
Data
Solu3ons
from
Red
Hat

Integrated
Big
Data
PlaOorm

Cisco UCS Integrated Infrastructure for Big Data
Hadoop
Compatible
File System
Red Hat
Storage
Hadoop Data Processing
Map/Reduce YARN
Analytics
Operating System
Red Hat Enterprise Linux
Cloud
Red Hat Enterprise Linux
OpenStack Platform
Operating EnvironmentData Integration & Application Development
Application Platform-
as-a-Service
OpenShift by Red Hat
Data Integration and Data
Services
Red Hat JBoss Data
Virtualization
Data Caching
Red Hat JBoss
Data Grid
Business Rules Mgmt
Red Hat JBoss BRMS
Development
Red Hat JBoss
Developer
Studio Hadoop
Distributed
File
System
Management
HortonworksCisco Red Hat
Data Integration
and Data
Services
Composite
Cloud
Cisco OpenStack
Pig Spark Storm
HBase Tez Hive
Cisco Security Suite
CiscoUCSDirectoryExpress
CiscoUnifiedManagement
Ambari
Virtualization
Red Hat Enterprise
Virtualization

Software and Solutions Innovation
Empowering What’s Next
Ron Graham
Big Data Analytics Engineer
Hardware Architecture
Cisco UCS with Big Data

20© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Why Cisco UCS for Big Data?
•  Manageability
•  Save time with UCS Manager
•  Enables consistent and rapid
deployments using UCS Service profiles
•  Offers operational simplification
•  Delivers a modular solution
•  Scalability
•  Performance
SIM Card
Identity for a phone
Service Profile
Identity for a server

•  End to end provisioning, installation, and
monitoring tool for Hadoop Clusters
•  Better business outcomes with faster time to
value from Big Data
•  Provides appliance like experience with out
inflexibilities
•  Centralized visibility across Hadoop and
physical infrastructure
•  Powerful interface for further integration into
third party tools and services
UCS Director Express for Big Data
End to end solution for Hadoop

Powering Big Data and Analytics
UCS
B200

Scale-‐out
Analy5cs

Big
Data

with

EMC

Isilon

and
VCE

Invicta

(Fast

Data)

UCS
C240

(Hadoop,
NoSQL

MPP)

UCS
Manager,
Director,
Express,
Central,
Redhat

ACl

C/B460
(In-‐
memory

Analy5cs)

UCS
C3160,

C3260

(Hadoop)

UCS
C220

(real-‐5me,
streaming)

FlexPod

Select

with

NetApp

E-‐Series
UCS
Mini
(All-‐in-‐one

at
Edge)

UCS
M-‐Series
(Massive

scale-‐out)

Ac5an,
DataStax,
Hortonworks,
MongoDB,
Pivotal,SAP,
SAS,
Splunk

Cisco,
Elas5c
Search,
IBM,
Informa5ca,
MicrosoZ,
MicroStrategy
,
Oracle,
SAP,

SAS

and
others

Complete

and
Industry

leading

Por[olio

Ecosystem

Partners

ISV
Partners

Infrastructure

Management

Data
Management

Applica5ons

DESIGNS
Big Data
Cisco Validated Designs
for leading big data
platforms can be found
at:
www.cisco.com/go/bigdata
Cisco Validated Designs
Accelerate Deployment

Server 8x UCS C220 M4
CPU 2 x Intel Xeon
E5-2620 v3 (15M
Cache, 2.40 GHz)
Memory 256GB
Storage 8 1.2-TB 10K SAS
SFF HDD
Starter High Performance
CPU 2 x Intel Xeon
E5-2680 v3 (30M
Cache, 2.50 GHz)
Memory 384GB
Storage 2 1.2-TB 10K SAS
SFF HDD, 6 400-
GB SAS SSD
Performance Optimized Capacity Optimized Extreme Capacity
CPU 2 x Intel Xeon E5-2680
v3 (30M Cache, 2.50
GHz)
Memory 256GB
Storage 2 120-GB SATA SSD,
24 1.2-TB 10K SAS
SFF HDD
CPU 2 x Intel Xeon
E5-2620 v3 (15M
Cache, 2.40 GHz)
Memory 128GB
Storage 2 120-GB SATA
SSD. 12 4-TB 7.2K
SAS SFF HDD
Server 2x UCS C3160
CPU 2 x Intel Xeon
E5-2695 v2 (30M
Cache, 2.40 GHz)
Memory 256GB
Storage 2 120-GB SATA
SSD, 60 4-TB 7.2K
SAS SFF HDD
Cisco UCS CPA for Big Data v3
Reference Architecture and Bundles

2x UCS 6296 Series
Fabric Interconnect
UCS Manager
•  UCS Domain (68 Servers)
•  Manage by UCS Manager
•  2.8 PB of storage
•  HDP 2.2
•  Tiered Storage
•  Tez
•  RHEL 6.5
•  Dual 10G Network
•  17 Servers Per Rack
UCS C240 M4
2x E5-2680 v3
256GB Memory
Cisco 12Gb/s SAS Raid Controller
2x 120GB STAT SSD
24x 1.2TB 10k SAS
2x Cisco UCS VIC 1227
UCS C3160
2x E5-2695 v2
256GB Memory
Cisco 12Gb/s SAS Raid Controller
2x 120GB SATA SSD
60x 4TB 7.2k SAS SFF
2x Cisco UCS VIC 1227
/ 17 10Gb Ethernet
/ 17 10Gb Ethernet
64 Node Cluster Configuration

UCSD Express
UCS 6200 Series
Fabric Interconnect
UCS Manager
UCS C240 M4 Series
Rack Server
UCS C3160 Rack
Server
Apache Ambari
Unified Management
Programmability, Scalability and Automation

UCS 6200 Series
Fabric Interconnect
UCS C240 M4 Series
Rack Server
UCS C3160 Rack
Server
Data
Data
Data
Cold
n replicas on
Archive
Warm
1 replicas on Disk,
n-1 on Archive
Hot
All (n) replicas on
Disk
Cold
Hot
Policy
Hot - for both storage and compute. The data that
is popular and still being used for processing will
stay in this policy. When a block is hot, all replicas
are stored in DISK.
Warm - partially hot and partially cold. When a
block is warm, some of its replicas are stored in
DISK and the remaining replicas are stored in
ARCHIVE.
Cold - only for storage with limited compute. The
data that is no longer being used, or data that
needs to be archived is moved from hot storage to
cold storage. When a block is cold, all replicas are
stored in ARCHIVE.
Multi-tiered Storage Architecture
Multi-temperature Policy

UCS 6200 Series
Fabric Interconnect
UCS C240 M4 Series
Rack Server
UCS C3160 Rack
Server
Data
Data
Data
Cold
n replicas on
Archive
Warm
1 replicas on Disk,
n-1 on Archive
Hot
All (n) replicas on
Disk
Cold
Hot
Mover – A new data migration tool
It periodically scans the files in HDFS to
check if the block placement satisfies the
storage policy. For the blocks violating the
storage policy, it moves the replicas to a
different storage type in order to fulfill the
storage policy requirement.
A
C
D
A
C
D
E
A
C
D
E
N
N
N
N
E
Multi-tiered Storage Architecture
Multi-temperature Policy

Next Steps…
Download the Hortonworks Sandbox
Learn Hadoop
Build Your Analytic App
Try Hadoop
Learn more with our partnerships
http://hortonworks.com/partner/cisco/
http://hortonworks.com/partner/redhat/
Joint CVD bit.ly/Cisco-CVD

•  Cisco Live! in San Diego – June 7 - 11
•  Hadoop Summit in San Jose – June 9 – 11
•  Red Hat Summit in Boston - June 23-26
More information about Red Hat’s Big Data solutions please visit:
•  redhat.com/bigdata
•  redhatstorage.redhat.com/category/big-data
•  redhat.com/en/insights/big-data
More information about Cisco’s Big Data and Analytics Offers please visit:
•  www.cisco.com/go/bigdata and www.cisco.com/go/bigdata_design
•  http://blogs.cisco.com/author/raghunathnambiar
•  bit.ly/Cisco-CVD
30
Meet us in person!

A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks and Red Hat

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks and Red Hat (20)

More from Hortonworks (20)

Recently uploaded (20)

A Comprehensive Approach to Building your Big Data - with Cisco, Hortonworks and Red Hat