Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (HDFS)

Discover HDP 2.2
Data Storage Innovations in Hadoop Distributed File System (HDFS)
Page 1 © Hortonworks Inc. 2014
Hortonworks. We do Hadoop.

Speakers
Rohit Bakhshi
Hortonworks Senior Product Manager & PM for Apache
Hadoop & Apache Solr in Hortonworks Data Platform
Jitendra Pandey
Hortonworks Senior Architect for HDFS

Agenda
• Overview of HDFS
• New HDFS Innovation in HDP 2.2
– Heterogeneous storage
– Encryption
– Operational security enhancements
• Q & A
We’ll move quickly:
• Attendee phone lines are muted
• Text any questions to Jitendra using Webex chat
• Questions will be answered at the end of the call
• Unanswered questions and answers in upcoming FAQ/blog post

Big Data, Hadoop & Data Center Re-platforming
Business Drivers
• From reactive analytics
to proactive interactions
• Insights that drive
competitive advantage
& optimal returns
$
Financial Drivers
• Cost of data systems, as
% of IT spend,
continues to grow
• Cost advantages of
commodity hardware
& open source software
Technical Drivers
• Data is growing
exponentially & existing
systems overwhelmed
• Predominantly driven by
NEW types of data that
can inform analytics
There is an inequitable balance between vendor and customer in the market

Clickstream
Capture and analyze
website visitors’ data
trails and optimize
your website
Sensors
Discover patterns in
data streaming
automatically from
remote sensors and
machines
Server Logs
Research logs to
diagnose process
failures and prevent
security breaches
Hadoop Value: New Types of Data
Sentiment
Understand how
your customers feel
about your brand
and products –
right now
Geographic
Analyze location-based
data to
manage operations
where they occur
Unstructured
Understand patterns
in files across millions
of web pages, emails,
and documents

A Shift from Reactive to Proactive Interactions
A shift in Advertising
From mass branding …to 1x1 Targeting
A shift in Financial Services
From Educated Investing …to Automated Algorithms
A shift in Healthcare
From mass treatment …to Designer Medicine
A shift in Retail
A shift in Telco
HDP and Hadoop allow
organizations to use
data to shift interactions
from…
Reactive
Post Transaction
Proactive
Pre Decision
…to Real-t From static branding ime Personalization
From break then fix …to repair before break

Enterprise Goals for the Modern Data Architecture
Batch Interactive Real-Time
• Consolidate siloed data sets structured
and unstructured
• Central data set on a single cluster
• Multiple workloads across batch
interactive and real time
• Central services for security, governance
and operation
• Preserve existing investment in current
tools and platforms
• Single view of the customer, product,
supply chain
DATA SYSTEM APPLICATIONS
Business
Analytics
Custom
Applications
Packaged
Applications
RDBMS
EDW
MPP
YARN: Data Operating System
1 ° ° ° ° ° ° ° ° °
°
° ° ° ° ° ° ° ° N
CRM
ERP
Other
1 ° ° °
° ° ° HDFS
(Hadoop Distributed File System)
SOURCES
EXISTING
Systems
Clickstream
Web
&Social
Geoloca9on
Sensor
&
Machine
Server
Logs
Unstructured

YARN Transformed Hadoop & Opened a New Era
Script
Pig
BATCH, INTERACTIVE & REAL-TIME DATA ACCESS
SQL
Hive
TezTez
YARN
The Architectural
Center of Hadoop
• Common data platform, many applications
• Support multi-tenant access & processing
• Batch, interactive & real-time use cases
Java
Scala
Cascading
Tez
Stream
Storm
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
Others
ISV
Engines
° ° ° ° °
° ° ° ° °
HDFS
Search
Solr
NoSQL
HBase
Accumulo
Sli der
Slider
In-Memory
Spark

YARN Extends Hadoop to Other Data Center Leaders
Script
Pig
SQL
Hive
TezTez
Java
Scala
Cascading
Tez
NoSQL
HBase
Accumulo
Sli der
1 ° ° ° ° ° ° °
Stream
Storm
Slider
HDFS
In-Memory
Spark
° ° ° ° ° ° ° °
YARN
The Architectural
Center of Hadoop
• Common data platform, many applications
• Support multi-tenant access & processing
• Batch, interactive & real-time use cases
• Supports 3rd-party ISV tools
(ex. SAS, Syncsort, Actian, etc.)
° °
° °
Others
ISV
Engines
Search
Solr
° ° ° ° °
° ° ° ° °
YARN Ready Applications
Facilitates ongoing innovation and enterprise adoption via
ecosystem of new and existing “YARN Ready” solutions

Enterprise Hadoop: Central Set of Services
GOVERNANCE SECURITY OPERATIONS
Tez
TezTez
Slider
Slider
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
° ° ° ° °
° ° ° ° °
Enables Apache Hadoop to be
an Enterprise Data Platform
with centralized services for:
• Governance
• Operations
• Security
Everything that plugs into
Hadoop inherits these services
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Load data and
manage
according
to policy
Deploy and
effectively
manage the
platform
Provide layered
approach to
security through
Authentication,
Authorization,
Accounting, and
Data Protection
Script
Pig
SQL
Hive
Java
Scala
Cascading
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
In-Memory
Spark
Others
ISV
Engines
HDFS

Hortonworks Development Investment for the Enterprise
Vertical Integration with YARN and HDFS
Tez
TezTez
Slider
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Slider
° °
° °
° ° ° ° °
° ° ° ° °
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Load data and
manage
according
to policy
Deploy and
effectively
manage the
platform
Provide layered
approach to
security through
Authentication,
Authorization,
Accounting, and
Data Protection
Script
Pig
SQL
Hive
Java
Scala
Cascading
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
In-Memory
Spark
Others
ISV
Engines
HDFS
• Ensure engines can run reliably and respectfully in a YARN based cluster
• Implement features throughout the stack to accommodate

Hortonworks Development Investment for the Enterprise
Horizontal Integration for Enterprise Services
Tez
TezTez
Slider
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Slider
° °
° °
° ° ° ° °
° ° ° ° °
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Load data and
manage
according
to policy
Deploy and
effectively
manage the
platform
Provide layered
approach to
security through
Authentication,
Authorization,
Accounting, and
Data Protection
Script
Pig
SQL
Hive
Java
Scala
Cascading
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
In-Memory
Spark
Others
ISV
Engines
HDFS
• Ensure consistent enterprise services are applied across the entire Hadoop stack
• Integrate with and extend existing data center solutions for these key requirements

HDP Delivers Enterprise Hadoop
Hortonworks Data Platform 2.2
GOVERNANCE BATCH, INTERACTIVE & REAL-TIME DATA ACCESS SECURITY OPERATIONS
Script
Pig
SQL
Hive
TezTez
Java
Scala
Cascading
Tez
Stream
Storm
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
° ° ° ° °
° ° ° ° °
HDFS
Search
Solr
NoSQL
HBase
Accumulo
Sli der
Slider
In-Memory
Spark
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
Kafka
NFS
WebHDFS
Authentication
Authorization
Audit
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive
Pipeline: Falcon
Cluster: Ranger
Cluster: Knox
Linux Windows Deployment Choice Cloud
YARN is the architectural
center of HDP
• Common data set across all
applications
• Batch, interactive & real-time
workloads
• Multi-tenant access & processing
Provides comprehensive
enterprise capabilities
• Governance
• Security
• Operations
Enables broad
ecosystem adoption
• ISVs can plug directly into Hadoop
The widest range of deployment options
• Linux & Windows
• On premises & cloud
Others
ISV
Engines
On-Premises

HDP Delivers Enterprise Hadoop
Hortonworks Data Platform 2.2
GOVERNANCE BATCH, INTERACTIVE & REAL-TIME DATA ACCESS SECURITY OPERATIONS
1 ° ° ° ° ° ° °
HDFS
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
TezTez
Java
Scala
Cascading
Tez
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Sli der
Slider
In-Memory
Spark
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
Kafka
NFS
WebHDFS
Authentication
Authorization
Audit
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive
Pipeline: Falcon
Cluster: Ranger
Cluster: Knox
YARN is the architectural
center of HDP
• Common data set across all
applications
• Batch, interactive & real-time
workloads
• Multi-tenant access & processing
Provides comprehensive
enterprise capabilities
• Governance
• Security
• Operations
Enables broad
ecosystem adoption
• ISVs can plug directly into Hadoop
° °
° °
° ° ° ° °
° ° ° ° °
The widest range of deployment options
• Linux & Windows
• On premises & cloud
Others
ISV
Engines
Linux Windows Deployment Choice On-Premises Cloud

Overview of HDFS

HDFS enables the Common Data Platform
Script
Pig
SQL
Hive
TezTez
HDFS
Storage Platform for Modern Data
Architecture
• Common data platform across multiple
application workloads
• Reliable
• Scalable
• Cost Efficient
Java
Scala
Cascading
Tez
Stream
Storm
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
Others
ISV
Engines
° ° ° ° °
° ° ° ° °
HDFS
Search
Solr
NoSQL
HBase
Accumulo
Sli der
Slider
In-Memory
Spark

HDFS Innovations on HDP 2.2

HDFS in HDP 2.2: What’s New
Heterogeneous
Storage
• Archive
and
SSD
Tiers
• Tech
Preview:
Enable
intermediate
data
to
stored
in
memory
Heterogeneous
Storage
THEME
Encryp9on
• Tech
Preview:
Transparent
Data
Encryp?on
Security
THEME
DataNode
does
not
require
Root
to
start
• HDFS
services
in
a
Kerberized
cluster
no
longer
need
Root
to
start
Security
THEME

New in HDP 2.2:
Heterogeneous Storage

Heterogeneous Storage
Before
• DataNode is a single storage
• Storage is uniform - Only storage type Disk
• Storage types hidden from the file system
New Architecture
• DataNode is a collection of storages
• Support different types of storages
– Disk, SSDs, Memory
All disks as a single storage
S3
Swift
SAN
Filers
Collection of tiered storages

HDFS Storage Architecture - Now

Storage Policies: Archival
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
ARCHIVE
ARCHIVE
ARCHIVE
ARCHIVE
ARCHIVE
ARCHIVE
ARCHIVE
ARCHIVE
ARCHIVE
Warm
1 replica on DISK,
others on ARCHIVE
Hot
All replicas on DISK
Cold
All replicas on
ARCHIVE
HDP Cluster

Storage Policy: SSD
SSD
DISK
DISK
SSD
DISK
DISK
SSD
DISK
DISK
SSD
DISK
DISK
SSD
DISK
DISK
HDP Cluster
A
SSD
DISK
DISK
A A
SSD
DataSet A All replicas on SSD

Store Intermediate Data in Memory
Application
Process
Write block to memory
Memory Tier
Lazy persist
block to disk
RAM_DISK
Tech Preview feature
For data writes that:
- Need low latency writes
- Where data is regenerate-able

New in HDP 2.2:
Encryption

HDFS Transparent Data Encryption
• HDFS Encryption – Transparent Encryption in HDFS – HDFS-6134
– Designate a dir as encryption zone, all files in the zone are encrypted
– Dependency on Key Management Server
• Key Management Server - HADOOP-10433
– The custodian for all encryption keys in Hadoop
– REST API for key CRUD operations
• Key Provider API - HADOOP-10141
– API to allow Hadoop code (NN, DN, DFS Clients) CRUD operations on key material

HDFS Transparent Data Encryption
1
°
°
°
°
1
°
°
°
°
°
Encrypted
File
(aIributes
-‐
EDEK,
IV)
°
°
°
°
°
°
Encryp9on
Zone
°
°
°
°
°
°
(aIributes
-‐
EZKey
ID,
version)
HDFS-‐6134
°
°
KeyProvider
°
°
°
°
Name
Node
°
°
°
°
N
DATA
ACCESS
DATA
MANAGEMENT
SECURITY
YARN
HDFS
Client
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
HDFS
°
(Hadoop
Distributed
File
System)
API
KeyProvider
API
KeyProvider
API
–
Hadoop-‐10141
Key
Management
System
(KMS)
Hadoop-‐10433
EDEK
DEK
Crypto
Stream
(r/w
with
DEK)
DEKs
EZKs
Acronym
Descrip?on
EZ
Encryp?on
Zone
(an
HDFS
directory)
EZK
Encryp?on
Zone
Key;
master
key
associated
with
all
files
in
an
EZ
DEK
Data
Encryp?on
Key,
unique
key
associated
with
each
file.
EZ
Key
used
to
generate
DEK
EDEK
Encrypted
DEK,
Name
Node
only
has
access
to
encrypted
DEK.
IV
Ini?aliza?on
Vector
EDEK
EDEK

New in HDP 2.2:
Operational Security Enhancements

DataNode does not require root
Enables Organizations to run services without utilizing root privilege
For Kerberized clusters
DataNode no longer needs to run as the Linux root user when starting
DataNode no longer needs to bind to privileged ports
DataNode utilizes SASL to transfer blocks between HDFS clients and
DataNodes.

Q & A

Thank you!
Learn more at:
hortonworks.com/hadoop/hdfs/
Register for the remaining 4
Discover HDP 2.2 Webinars
Hortonworks.com/webinars

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (HDFS)

Recommended

More Related Content

What's hot (20)

Viewers also liked (17)

Similar to Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (HDFS) (20)

More from Hortonworks (20)

Recently uploaded (20)

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (HDFS)