HBaseCon 2013: Multi-tenant Apache HBase at Yahoo!

Multi-tenant Apache HBase at Yahoo!
Francis Liu, Sumeet Singh
HBaseCon, June 13, 2013

Introduction
Yahoo! Presentation, Confidential
2
Sumeet Singh
Head of Products, Cloud Services &
Hadoop
Cloud Engineering Group
701 First Avenue
Sunnyvale, CA 94089 USA
Francis Liu
Principal Software Engineer, Hadoop
Cloud Engineering Group
§  Leads Apache HBase development at Yahoo!
§  Apache Hive contributor
§  Apache HCatalog committer
§  Developed a workflow management and
incremental processing platform
701 First Avenue
Sunnyvale, CA 94089 USA
§  Manages Cloud Services and Hadoop products
teams at Yahoo!
§  Responsible for Product Management, Customer
Engagements, Evangelism, and Program
Management
§  Pretends to be the Product Manager for Hadoop
at Yahoo!
§  Headed Strategy for Cloud Platform Group at
Yahoo!

Agenda
Yahoo! Presentation, Confidential 3
Apache HBase and Motivation for Multi-tenancy1
Apache HBase Use Cases at Yahoo!2
Integration with Other Apache Hadoop Projects3
Accomplishing Multi-tenancy in Apache HBase4
Experiences in deploying secure Apache HBase5

Multi-tenant Apache Hadoop at Yahoo!
0
50
100
150
200
250
300
350
400
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
40,000
45,000
2006 2007 2008 2009 2010 2011 2012
RawHDFSStorage(inPB)
NumberofNodes
Year
Nodes HDFS
Yahoo!
Commits to
Scaling Hadoop
for Production
Use
Research
Workloads
in Search and
Advertising
Production
(Modeling)
with machine
learning &
WebMap
Revenue
Systems with
Security, Multi-
tenancy, and
SLAs
Increased
User-base
with partitioned
namespaces
Open
Sourced with
Apache
Hortonworks
Spinoff for
Enterprise
hardening
Current
Team with
Y! focus
Footprint: 42,000 nodes, 365PB HDFS, 10M daily slot hours | Usage (Apr’13): 411 projects, 805 users, 13.2M jobs
4

HBase at Y! – Multi-tenancy at the Helm Again
0
1
2
3
4
5
0
500
1,000
1,500
2,000
2,500
3,000
2009 2010 2011 2012 2013
DataStored(inPB)
NumberofNodes
Year
Nodes Data Stored
Content
Personalization
Web Crawl
Cache
Grid Sandbox
Personalization
& Targeting
Phase I
Private deployments
Phase III
Prod. Multi-tenancy
Phase II
Grid Sandbox
5
Projected Footprint by 2013: 2,400 nodes, 4.6PB of data stored | Projected Usage by 2013: 18 projects
16 new projects
(4 on-boarded, 12 upcoming)

HBase Covers Most of Yahoo!’s Businesses
Yahoo!’s Global Businesses
Search
§  Web Cache
§  Query Analysis
§  Local Listings
§  Analytics
Y! Mail
§  Anti-spam
§  Log Analytics
§  Metadata Mgmt.
Cloud Platforms
§  Performance
§  Monitoring
§  OpenStack
Consumer Platforms
§  CMS
§  Social Data
Online Ads
§  Traffic Protection
§  Ads Data Mgmt.
P13N
§  Content P13N
§  Ad targeting
Mobile
§  Notifications
§  Flickr
Sales
§  eCommerce
6

The Appeal of Apache HBase to Yahoo!
Yahoo! needed a solution to store mutable data and support random access.
HBase was an obvious choice.
§  Native to the Hadoop ecosystem
§  Large Hadoop developer base
§  Integration with other Hadoop components popular at Yahoo! such as MapReduce, Pig, Oozie,
HCatalog, Hive
§  Vibrant/ active open source community of developers
§  Attractive throughputs, in particular, the write throughput
§  Acceptable latencies and scan performance
§  Support for Yahoo!’s scale
§  Support for bulk uploads
§  Easy on-boarding to the multi-tenant platform
§  Easy application development (lower time to market)
§  Support for dynamically adding columns, TTL, versioning and timestamps
7

Use Case 1 – Content Personalization (V1)
Y! Property
Server
Audience Events
User Activity
(User Pipeline)
Content
Content, Rules
(Content Pipeline)
Models
Index
Serving Scores
(User/ Content)
User Profiles | Content Index
8
Event
Collection
Content
Management
System
Serving
Systems
HDFS
MapReduce

Use Case 1 – Content Personalization (V2)
Y! Property
Server
Audience Events
User Activity
(User Pipeline)
Content
Content, Rules
(Content Pipeline)
Serving Scores
(User/ Content)
User Profiles | Content Index
9
Event
Collection
Content
Management
System
Serving
Systems
Storm
(Short-term
Pipeline)Models
Index
HDFS
MapReduce

HBase Helps Yahoo! Deliver Personalization

Poller
Use Case 2 – Web Crawl Cache
Fetcher
Ingestor Extruder
Processing
Cluster
Random
Read
End Users
Consumption
poll
fetch
launch
launch
write
read,
write
scan

Use Case 3 – Detecting Abusive Accounts
Outgoing Mail Server
Anti-spam
Rules Engine
User Feedback
Serving Maps
Machine Learning
Spam Detection Models
Spam
Patterns
Categorization
Models
Good, Offer CAPTCHA, Block
Near Real-time
Classifier
Compromised Acct.
Force pwd. Change/
deactivate acct.
Incoming Mail Server
Spam Filtering Abusive Account Detection

Use Case Pattern 1 – Persist Metadata/ State
write read
CF – MessageStore CF - Attributes
Rowkey MessageId Payload Colo
MD5 Hash
Mailbox Id # 1
1 “...” DC1
MD5 Hash
Mailbox Id # 2
2 “...” DC2
MD5 Hash
Mailbox Id # 3
3 “...” DC3

Use Case Pattern 2 – Metrics/ Analytics
Collector Collector Collector
Ingestion
Query Server

Use Case Pattern 3 – Dimension Store
MapReduce
Hive
Pig
Clickstream Ad Campaign

Use Case Pattern 4 – Incremental Processing
Collector
MapReduce
Storm
Serving
Stores
Batch (Long-term) Pipeline
Near Real-time (Short-term) Pipeline
Collection Off-stage Processing On-stage Serving
Files
Events
Search
Index

Use Case Patterns Summary
Use Case Persist Metadata/ State Metrics/ Analytics Dimension Store Incremental Processing
Search
Web Cache ✔
Query Analysis ✔
Local Business Listings ✔
Analytics ✔
Y! Mail
Anti-spam ✔
Log Analytics ✔
Metadata Management ✔
Cloud Platforms
Performance ✔
Online Monitoring ✔
OpenStack ✔
Consumer Platforms
Content Management ✔
Social Data ✔
Online Ads
Traffic Protection ✔
App Data Management ✔
P13N
Content Personalization ✔
Ad Targeting ✔
Mobile
Notifications ✔
Sales
eCommerce ✔

Integration With Other Hadoop Components
TableInputFormat
TableOutputFormat
HBaseStorage
HandlerHBaseStorage HBase Credential
Read/Write with MR Read/Write into existing tables Create/Drop/ Read/Write Secure HBase access

Apache Hadoop at Yahoo!
§  Hosted Multi-tenant Service
§  Security
§  Job Queues
§  HDFS Quota

Apache HBase at Yahoo!
§  Hosted Multi-tenant Service
§  Security
§  Isolated Deployment
§  Region Server Group (HBASE-6721)
§  Namespace (HBASE-8015)

Security
§  Authentication
§  Kerberos (users, processes)
§  Delegation Token (MapReduce, YARN, etc.)
§  Authorization
§  HBase ACLs (Read, Write, Create, Admin, Exec)
§  Grant permissions to User or Group
§  ACL for Table, Column Family or Column
§  Only Global Admin can create/drop tables

Security at Yahoo!
§  Used in a trusted environment
§  Prevent misuse vs. malicious use
§  Privacy
§  Auditing
§  Deployed in production for the last 8 months
§  No major issues

Security Issues
§  Unprotected APIs
§  bulkLoad (HBASE-5498)
§  stopRegionServer, regionClose, regionOpen (HBASE-7331)
§  WebUI
§  Compact & Split
§  Logs
§  Client Authentication not honored
§  Use hadoop-policy.xml
§  REST & Thrift interfaces have no authentication

Security Bugs
§  stopMaster (HBASE-7066)
§  Kerberos Replay
§  Ecosystem integration (Oozie, Pig, etc.)
§  Refreshing Proxy Users
§  Delegation Token (HBASE-6671, HBASE-7771, HBASE-7772)

Isolated Deployment
HBase
Client
HBase
Client
JobTracker Namenode
TaskTracker
DataNode
Namenode
RegionServer
DataNode
RegionServer
DataNode
RegionServer
DataNode
HBase MasterZookeeper
Quorum
HBase
Client
MR Client
M/R Task
TaskTracker
DataNode
M/R Task
TaskTracker
DataNode
MR Task
Compute Cluster HBase Cluster
Gateway/Launcher

Region Server Groups Overview
§  Member Region Servers
§  Member Tables
§  Resource Isolation
§  Flexibility with configuration
Group Bar
Region Server 5…8
Table3
Table4
Group Foo
Region Server 1…4
Table1
Table2
RS1
Table1
Table2
RS2
Table1
Table2
RS3
Table1
Table2
RS4 RS5
Table3
Table4
RS6
Table3
Table4
RS7
Table3
Table4
RS8

Region Server Groups APIs
§  group_add
§  group_remove
§  group_move_servers
§  group_move_table
§  group_balance
§  create … { …
CONFIGURATION=>{‘hbase.rsgroup.name’=>’my_group’}}
§  Requires Global Admin Privileges

Region Server Groups Implementation
LoadBalancer
GroupBasedLoadBalancer
GroupAdminEndpoint
GroupMasterObserver
HMaster
FilterBy
Group
foo
bar
GroupInfoManager
Group Table
Group
ZNode

Namespace
§  Analogous to Database
§  Full Table Name: <table namespace>.<table name>
§  i.e. my_ns.my_table
§  Reserved namespaces
§  Default – tables with no explicit namespace
§  System – tables are guaranteed to be assigned prior to user
tables
§  Table Path: /<hbaseRoot>/.data/<namespace>/
<tableName>
§  /hbase/data/my_ns/my_ns.my_table

Namespace + Security + Quota + Group
Namespace
Group Tables Quota ACL

Namespace + Security + Quota + Group
§  Default Region Server Group
§  Write privilege for table creation/deletion
§  Quota
§  Max Tables
§  Max Regions
§  Per Tenant
Namespace
Group Tables Quota ACL

Namespace + Quota
HMaster
TableNamespaceManager
Namespace
Table
Namespace
ZNodes
Namespace NamespaceController
ZKNamespaceManager
MasterCPHostRegionCPHost

Conclusion
§  Security
§  Isolation
§  Isolated Deployment
§  Region Server Groups
§  Resource Allocation
§  Namespace

HBaseCon 2013: Multi-tenant Apache HBase at Yahoo!

HBaseCon 2013: Multi-tenant Apache HBase at Yahoo!

Recommended

More Related Content

What's hot (20)

Similar to HBaseCon 2013: Multi-tenant Apache HBase at Yahoo! (20)

More from Sumeet Singh (15)

Recently uploaded (20)

HBaseCon 2013: Multi-tenant Apache HBase at Yahoo!