How data modelling helps serve billions of queries in millisecond latency with real-time streaming

© 2019 Bloomberg Finance L.P. All rights reserved.
How data modelling helps serve
billions of queries in millisecond
latency with real-time streaming
NoSQL Day 2019 at DataWorks Summit DC
May 21, 2019
Udai Bhan Kashyap, Senior Software Engineer (Equities)
Amit Anand, Senior Software Engineer (Hadoop Infrastructure)

© 2019 Bloomberg Finance L.P. All rights reserved.© 2019 Bloomberg Finance L.P. All rights reserved.
Agenda
About Bloomberg
Problem Definition
HBase and Data Modeling
Q&A

Bloomberg in nutshell
Image Source: Bloomberg

Bloomberg by numbers
Founded in 1981
325,000 subscribers in 170+ countries
Over 19,000 employees in 192 locations
More News reporters than The New York Times + Washington Post + Chicago
Tribune

Bloomberg Tech
5,500+ Software engineers and growing
100+ engineers and data scientists devoted to machine learning
One of the largest private networks in the world
100B+ market data ticks per day, peak of 10 million+ messages/sec
News content from 125K+ sources
2M news stories each day (that's >500 news stories ingested/sec)
More than a billion emails and IB chats handled each day

Serve billions of queries with low latency, transactional consistency, and real-
time streaming
Problem Definition

Fast access and availability of the data is key for users and
automated systems due to the market-impacting nature of the data
Wide variety of applications (e.g., Quantitative Analysis,
Backtesting, Screening, etc.) require huge amount of data,
demanding a high throughput and low latency system
Low Latency
Image source: Pixabay

Real-Time Streaming

Our data/analytics is
used by traders, fund
managers, educational
institutions …
And it impacts the
market!
Image source: Bloomberg

Transactional Consistency
Google’s 2018 Annual Filing

The numbers
together tell the
story

“Revenue” = “Sales & Services Revenue” + “Other Revenue”
“Gross Profit”= “Revenue” - “Cost of Revenue”

HBase

Ordered Key-Value Store
Distributed - provides horizontal scaling
Low Latency Random Access – necessary for interactive/real-time systems
Auto Sharding – operational ease!
HBase

ZooKeeper
Region Server
Region
Region
Region
Region
Region Server
Region
Region
Region
Region
Region Server
Region
Region
Region
Region
ZooKeeper ZooKeeper
HDFS Storage
HBase
Master
(Backup)
HBase
Master
(Active)
HBase Architecture

Data Modeling

Image Source: http://www.spatialdbadvisor.com/data_models
Data Modeling - RDBMS

RowKey – analogous to Index in RDBMS
Column Family
Column
HBase Table - Main components

Distribution

● Pre-Splitting Regions
○ Avoids hotspotting
○ Helps in initial mass on-boarding of the data
○ Minimizes split after deployment
Distribution - Be in control

● Key Distribution
○ Keys should be distributed (not related keys)
○ Identify components of the key that require
distribution
○ Generate hash based on identified components
○ Prepend the hash to your key
○ Do not allow region split to occur for the key
components involved in distribution. For example, in
our case, we won’t let IBM’s data be distributed
across multiple regions
Distribution - Be in control

Key/Index Design
Right key is important
HBase supports only one key on a table
Lexicographically sorted
Unlike RDBMS, key doesn’t need to be part of columns
Sequence of bytes

Design logical composite key as opposed to hash key
Logical key allows predicates on key’s components
Logical key preserves locality of related data
Only “Get” APIs can be used on hash key
Design small keys
Key/Index Design
Image source:Pixabay

What should be in the Key?
Image sources:Pixabay

Data items that require predicates
Data items that do not require atomic updates
What should be in the Key?

HBase doesn’t demand schema for table, but column families are tied to the
table and must be provided at table creation
Unlike RDBMS, columns are tied to Row(Key) and column sets can be unique
for each row
While there is no name for the key, the columns and column families are
named using ASCII characters
Avoid long names for columns and column families
Column Family and Columns

Column Family and Column names become part of the RowKey and are
carried with every data point
Column Family and Columns
Key-1 V2 Vn
Key-1 CF1:CQ1 V1
RDBMS V1
HBase
Key-1 CF1:CQn Vn

Grouping of related data items
Each column family has its own storage file
Column Family

Analyze your dataset to identify different patterns (e.g., sizes and/or queries)
Each pattern can be a column family
Identifying Column Families

What should be columns?

Data items requiring atomic updates or transactional behavior should be
columns
Last component of logical composite key can be a column
Key-1 CF1:CQ1 V1
Key-1 CF1:CQn Vn
What should be columns?

Why are we doing all this?

Atomicity – An operation completely succeeds or completely fails
Consistency – A successful mutation is visible to all read clients
Isolation – Operations do not interfere with each other
Durability – A successful mutation shall not be lost (even after system crash)!
HBase supports these properties at the Row level
https://hbase.apache.org/acid-semantics.html
ACID and HBase

RDBMS vs. HBase
Unlimited
# Rows
Fixed # Columns
Unlimited
# Rows
Varying and Unlimited # Columns
RDBMS HBase

d1+d2+..dn CF1:C1 value
d1+d2+..dn CF1:C2 value
d1+d2+..dn CF1:Cn value
HBase
D1 D2 … Dn C1 C2 … Cn
D1 D2 … Dn
d1 d2 … dn c1 c2 … cn
RDBMS
RDBMS vs. HBase

Data flow architecture
Publishing
Queue
Transformation
HBase
Middleware
Publishing
Publishing
Transformation

RETRIEVER
UPDATERS
QueueCluster
HBaseCluster
MSG Set
ACK
Groups of
MSGs
ACKS
Waiting
for ACKs
Apply
Changes
Reads
Updates
Streaming task architecture

Thank you!
We are Hiring!!
https://www.bloomberg.com/careers
Questions?

How data modelling helps serve billions of queries in millisecond latency with real-time streaming

Recommended

More Related Content

What's hot (20)

Similar to How data modelling helps serve billions of queries in millisecond latency with real-time streaming (20)

More from DataWorks Summit (20)

Recently uploaded (20)

How data modelling helps serve billions of queries in millisecond latency with real-time streaming