High Scale Relational Storage at Salesforce Built with Apache HBase and Apache Phoenix

High Scale Relational Storage at
Salesforce built with Apache HBase
and Apache Phoenix
 Andrew Purtell
 Architect, Cloud Storage
apurtell@salesforce.com
 @akpurtell
 
v3

 Safe harbor statement under the Private Securities Litigation Reform Act of 1995:
 This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties
materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed
or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-
looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any
statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new,
planned, or upgraded services or technology developments and customer contracts or use of our services.
 The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new
functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our
operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of any
litigation, risks associated with completed and any possible mergers and acquisitions, the immature market in which we operate, our
relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our
service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger
enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our
annual report on Form 10-K for the most recent fiscal year and in our quarterly report on Form 10-Q for the most recent fiscal quarter.
These documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section
of our Web site.
 Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available
and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features
that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.
Safe Harbor

 Architect, Cloud Storage at Salesforce.com
•  Data Platform Team
 
Open Source Contributor, since 2007
•  Committer, PMC, and Project Chair, Apache HBase
•  Committer and PMC, Apache Phoenix
•  Committer, PMC, and Project Chair, Apache Bigtop
•  Member, Apache Software Foundation
Distributed Systems Nerd, since 1997
whoami

 Motivation
 Open Source and the Data Platform Team
 What are Apache HBase and Apache Phoenix?
 HBase@Salesforce
•  The View from 30Kft
•  Keeping Up Appearances
•  Engineering The Whole Stack Holistically
 Q&A
Agenda

The Data Management Challenge
Scale data requires a single platform for analysis
(“data gravity”)
Salesforce already manages over 100B records of
customer data today
More than 3 billion transactions per day on the
platform
Exponential growth rate
 Batch and stream compute data locality
requirements
 Service continuity requirements

 Systems of record provide reliable, highly
available, and secure data storage
•  SObjects: Traditional Salesforce platform objects
•  Salesforce Files: Blob storage
•  BigObjects: Scale out storage for immutable data
BigObjects
Systems of Record
•  Transactional data. Rows can be added and
updated
•  Example: Accounts, Contacts, Custom Objects
SObjects
BigObjects
Salesforce
Files
•  New Object type for immutable data.
•  Optimized for large volumes of data
•  Example: event data, purchase history, product
usage data
•  Blob storage for semi- or un-structured data
•  Example: CSV extracts from external systems,
weblogs, monitoring logs
Platform Connect
External Object
•  New proxy object connected to an external
oData source

Data Pipelines
Snapshot data from
SObjects / External Objects
Manipulate and analyze data
sets
•  Transformations
•  Joins
•  Calculations and enrichment
Data
Snapshot
External
Objects
Salesforce
Files
BigObjects
Data
Pipeline

Data Pipelines
Snapshot data from
SObjects / External Objects
Manipulate and analyze data
sets
•  Transformations
•  Joins
•  Calculations and enrichment
Apache Pig + Hadoop for
processing framework and
scripting language
Apache HBase for BigObjects
persistence
Apache Phoenix for indexing
and relational access
Data
Snapshot
External
Objects
Salesforce
Files
BigObjects
Data
Pipeline

Data Management Services
Customer 360
Enrich customer
proﬁle w/ data from
external systems
Audit & TrackingData Retention Data Quality Connected
Products
Track what users are
doing on platform
Compliance
Keep data for audit
purposes
Maintain scalability
of operational
systems
Data enrichment
Data integration
Data cleansing
Event stream archive
Oﬄine analysis and
data mining

Salesforce Shield
Persist the complete ﬁeld
change history for up to ten
years with Field Audit Trail
Persist user event capture for
Event Monitoring and Audit
Support scale out batch and
stream compute for Backup,
DR, Threat Detection, etc.
Infrastructure Services
Network Services
Application Services
Secure
Data
Centers
Backup and
Disaster
Recovery
HTTPS
Encryption
Penetration
Testing
Advanced
Threat Detection
Identity &
Single Sign On
Two Factor
Authentication
User Roles &
Permissions
Field & Row
Level Security
Secure
Firewalls
Real-time
replication
Password
Policies
Third Party
Certiﬁcations
IP Login
Restrictions
Customer
Audits
Salesforce Shield
Platform
Encryption
Event
Monitoring
Field Audit Trail

Open Source and the Data Platform Team

The things we build are better
•  Building software with Open Source in mind is inherently beneﬁcial (encourages loose coupling,
cohesiveness, quality)
•  Positive external pressure to make the right software engineering decisions
 The things we don't build are better
•  The only thing better than writing a great component is not writing it
•  Smart engineers avoid “Not-Invented-Here” syndrome
 We are happier
•  Open Source extends pride in ownership and visibility beyond the company walls
•  We attract better coders who gravitate towards companies that do open source work
 We make the world a better place
How Open Source Helps Salesforce

 No forking
•  Fork defined as: a departure from the open source repository significant enough to prevent us from
contributing patches back or applying patches from the upstream open source repository
 Internal repositories as change buffer with local release numbering
•  Local repos for HBase, Phoenix, all dependencies (Hadoop, ZooKeeper, etc.)
•  Fast forwarded to new upstream releases after consideration
•  Updates to local repos are all manual by design
•  Hadoop is a special snowflake
 Change the upstream repositories first
•  Almost all changes begin as patches developed using upstream repositories
•  Only critical bug fixes are an exception, or where we require a change locally meantime while working with
a slow moving upstream community
How We Contribute To And Use Open Source

 Distributed systems engineers
 Storage systems architects
 Open source contributors
•  Project Chairs: Apache HBase, Phoenix, Bigtop
•  Committers and PMC: Apache HBase, Phoenix, Pig, Bigtop, Incubator
•  Mentors: Apache Phoenix, NiFi, Trafodion
•  Hundreds of commits per year
Who We Are

Apache HBase and Apache Phoenix
A new scale out relational storage option

 A high performance horizontally scalable datastore engine for Big Data suitable as the store of
record for mission critical data
Apache HBase

 An emerging platform for scale out relational datastores
Apache HBase
Apache Kylin

Apache HBase
 A founding member of the
Hadoop pantheon
•  Introduced as a Hadoop ‘contrib’
module in 2007

 Tablespaces
 Not like a spreadsheet, a “sparse, consistent, distributed, multi-dimensional, sorted map”
HBase Data Model

HBase Scalability
RegionServers
Table A
Table B
Splits
Assignments
Regions

How is HBase Diﬀerent from a RDBMS?
RDBMS HBase
Data layout Row oriented Column oriented
Transactions Multi-row ACID
Single row or adjacent row
groups only
Query language SQL None (API access)
Joins Yes No
Indexes On arbitrary columns Single row index only
Max data size Terabytes Petabytes*
R/W throughput limits
1000s of operations per
second
Millions of operations per
second*
* - No architectural upper bound on data size or aggregate throughput

 1969: CODASYL (network database)
 1979: First commercial SQL RDBMs
 1990: Transaction processing on SQL now popular
 1993: Multidimensional databases
 1996: Enterprise Data Warehouses
 2006: Hadoop and other “big data” technologies
 2008: NoSQL
 2011: SQL on Hadoop
 2014: Interactive analytics on Hadoop and NoSQL with SQL
 Why?
SQL: In and Out of Fashion
From “SQL On Everything, In Memory” by Julian Hyde, Strata NYC 2014

 Implementing structured queries well is hard
•  Systems cannot just “run the query” as written
•  Relational systems require the algebraic operators, a query planner, an optimizer, metadata, statistics, etc.
 However, the result is very useful to non-technical users
•  Dumb queries (e.g. tool generated) can still get high performance
•  Adding new algorithms or reorganizations of physical data layouts or migrations from one data store to
another are transparent
 What about blending the scale and performance of non-relational scale out stores with the ease
of use of SQL?
SQL: In and Out of Fashion

 A relational system built on HBase
•  Reintroduces the familiar declarative SQL interface to data
•  Reintroduces typed data and query optimizations possible with it
•  Secondary indexes, joins, query optimization, statistics, …
•  Integrates with just about everything as a JDBC data source
 With some new advantages
•  Dynamic columns extend schema at runtime
•  Schema is versioned – for free by HBase – allowing ﬂashback queries using prior versions of metadata
 And new challenges
•  Distributed transactions are hard; a work in progress building on Tephra*
“Putting the SQL back into NoSQL”
Apache Phoenix
* - Tephra open source transaction engine: http://blog.cask.co/2014/07/meet-tephra-an-open-source-transaction-engine-2/

Apache Phoenix
Supported?
CREATE / DROP / ALTER TABLE Yes
UPSERT / DELETE Yes
SELECT Yes
WHERE / HAVING Yes
GROUP BY / ORDER BY Yes
LIMIT Yes
JOIN Yes, with limitations
Views Yes
Secondary indexes Yes
Statistics and query optimization Yes
Transactions No, work in progress

Phoenix Integration With HBase
RegionServers
Regions
JDBC
driver
Coprocessors
(HBase server
side extensions)
SELECT * FROM users WHERE
id = ‘xxxabc’

 Phoenix maps the HBase data model to the relational world
Phoenix Data Model

Phoenix Data Model
Phoenix table

Phoenix Data Model
Phoenix table
Primary key constraint

Phoenix Data Model
Phoenix table
Primary key constraint Columns

Phoenix Data Model
Phoenix table
Primary key constraint Columns
Multiple
versions

 Dynamic columns
•  Specify a subset of columns at CREATE time; the remainder can be optionally speciﬁed at query time
•  Surfaces HBase’s schema ﬂexibility
Extras
CREATE TABLE “t” (
K VARCHAR PRIMARY KEY,
“f1”.”col1” VARCHAR);
SELECT * FROM “t” (“f1”.”col2” VARCHAR);
Dynamic column

 Native multitenancy
•  Multitenant isolation via a combination of multitenant tables and tenant-specific connections
•  Tenant-specific connections only access data that belongs to the tenant
•  Tenants can create their own schema addendums (views, columns, indexes)
Extras
CREATE TABLE event (
tenant_id VARCHAR,
type CHAR(1),
event_id BIGINT,
…
CONSTRAINT pk PRIMARY KEY (tenant_id, type, event_id))
MULTI_TENANT=true;
First PK column identifies tenant ID
Tenant-specific connection
DriverManager.connect(“jdbc:phoenix:localhost;tenantId=me”);

 Three index types
•  Mutable indexes
•  Global mutable indexes
 Server side intercepts primary table updates, builds and writes entries to secondary index tables
 For read heavy, low write uses cases
•  Local mutable indexes
 Index data and primary data are placed together (index in shadow column family)
 For write heavy, space constrained use cases
•  Immutable indexes
 Managed entirely by the client, writes scale best
 For use cases where rows are immutable after write
Secondary Indexes

 …
•  Global mutable indexes – covered indexes
•  Data in covered columns will be copied into the index
•  This allows a global index to be used more frequently, as a global index will only be used if all columns
referenced in the query are contained by it
Note: Rebuilds must currently be done using oﬄine tooling (MapReduce)
Secondary Indexes
CREATE TABLE t (k VARCHAR PRIMARY KEY,
v1 VARCHAR, v2 INTEGER);
CREATE INDEX i ON t (v1) INCLUDE (v2);
Covered column

 Standard join syntax is supported, with some limitations
Joins
•  Only equality (=) is supported in joining
conditions
•  No restriction on other predicates in the ON
clause
•  Some queries may exceed server side
resources and require rewrites by hand
•  Work in progress

 Client side rewriting
•  Parallel scanning with ﬁnal client side merge sort
•  RPC batching
•  Use secondary indexes if available
•  Rewrites for multitenant tables
 Statistics
•  Use guideposts to increase intra-region
parallelism
 Current Work-in-Progress
•  Integration with Apache Calcite
•  ~120 rewrite rules
Query Optimization
 Server side push down
•  Filters
•  Skip scans
•  Partial aggregation
•  TopN
•  Hash joins

Query Optimization
Example query plan for a 32 region table

Query Optimization
With a secondary index on RESPONSE_TIME

 Durability
 Consistency
 Modeling constructs
 Atomicity
 Concurrency
 Queryability
 Schema mutability
 Data size
 Data portability and backup
 Scalability
 Reliability
 Latency
 Recoverability
 Data Integrity
 Required Downtime
 Manageability
 Cost
 Support
 Comprehensibility
 License
 Community
So Why HBase?
The View from 30Kft

 Durability
 Consistency
 Atomicity
 Concurrency
 Queryability
 Data size
 Scalability
 Reliability
 Latency
 Recoverability
 Data Integrity
 Manageability
 Cost
 Support
 License
 Community
So Why HBase?
The View from 30Kft
 Scalability
 Consistency
 Operations
 License
 Community

 Durability
 Consistency
 Atomicity
 Concurrency
 Queryability
 Data size
 Scalability
 Reliability
 Latency
 Recoverability
 Data Integrity
 Manageability
 Cost
 Support
 License
 Community
So Why HBase?
The View from 30Kft
 Scalability
 Consistency
 Operations
 License
 Community
 HBase

 Instances are identical collections of hardware and software that support a discrete subset of our
customers
 An instance group is a collection of instances, in one location, configured as a single resource
pool and failure domain
 For every instance or instance group, there is an identical mirror available in another datacenter
 HBase is currently operating both in instance and instance group configurations
•  We started with smaller configurations (~10s of nodes)
•  Recently we have begun migrating to larger configurations (~100s of nodes)
•  Total fleet size is a “few thousand” servers
Instances and instance groups
The View from 30Kft

 Apache Phoenix was motivated by the impedance mismatch between the HBase API and the
expectations of platform developers
Keeping Up Appearances
SELECT * FROM foo WHERE bar > 30 HTable t = new HTable(“foo”);
RegionScanner s = t.getScanner(new Scan(…,
new ValueFilter(CompareOp.GT,
new CustomTypedComparator(30)), …));
while ((Result r = s.next()) != null) {
// blah blah blah Java Java Java
}
s.close();
t.close();
vs.

Engineering The Whole Stack Holistically

Kernel tunables, ﬁlesystem, krb, nscd

GC tuning, local patches

Multi-standby (HDFS-6440), fsync,
tuning, custom UGI

Many bug ﬁxes and enhancements,
tuning
tuning, custom UGI

tuning
Relational engine built on HBase
tuning, custom UGI

tuning
Data access layer for internal users
tuning, custom UGI

tuning, custom UGI
tuning
Data access layer for internal users
Circuit breaker, relogin thread

High Scale Relational Storage at Salesforce Built with Apache HBase and Apache Phoenix

Recommended

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to High Scale Relational Storage at Salesforce Built with Apache HBase and Apache Phoenix (20)

More from Salesforce Engineering (20)

Recently uploaded (20)

High Scale Relational Storage at Salesforce Built with Apache HBase and Apache Phoenix