Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East talk by Ruhollah Farchtchi

Building realtime BI
Systems with Kafka,
Spark and Kudu
Ruhollah Farchtchi
Zoomdata

Drivers for Streaming Data
Data Freshness Time to Analytic Business Context
2

Streaming Data @ Zoomdata
Visualizations react to
new data delivered
Users start,
stop, pause
the stream
Users select a rolling
window or pin a start
time to capture
cumulative metrics
3

Typical Streaming Architectures
Event
Kafka, JMS,
RabbitMQ,
etc...
Spark
Streaming,
Flink, etc..
Now What?
Cassandra (no
aggregation)
HDFS (what
about query)
Lambda (let’s
take a look at
that for a sec)
4

Lambda
● Stream data to hdfs
● Keep some in avro
● Do your own compactions to
parquet / orc
● Expose via impala,
sparksql, or other
● Impala avro partition
(speed)
● With history in parquet
● Union compatible schema
● Project as single table via
view
● Works ok… still doing a lot
of manual data
management
Oh.. and what happens to noncommutative operations like
Distinct Count?
OR
5

Restatements … yeah we went there
Txn ID Item Price Quantity Partition
1 Jeans 25.00 2 2016-05-30
2 Shirt 10.00 1 2016-05-31
3 Skirt 20.00 1 2016-06-01
1 Jeans 25.00 3
Restatement
6

Restatements … how you do it
Txn ID Item Price Quantity Partition
1 Jeans 25.00 2 2016-05-30
2 Shirt 10.00 1 2016-05-31
3 Skirt 20.00 1 2016-06-01
1 Jeans 25.00 3
Restatement
General Algorithm
● Figure out which partition(s) are affected
● Recompute affected partition(s) with restated data
● Drop/replace existing partition(s) with new data
7

Enter Kudu
What is Kudu?
● Kudu is an open source storage engine for structured data which supports low-latency
random access together with efficient analytical access patterns. (Source: http://getkudu.io/kudu.pdf)
Why do you care?
● It makes management of streaming data for ad-hoc analysis MUCH easier
● Bridges the mental gap from random access to append only
Why does Zoomdata care?
8

Impala + Kudu: Performance
Nearly the same performance as Parquet for many similar workloads
Simplified data management model
Can handle a new class of streaming use cases and workloads
9

Impala + Kudu: Performance
Nearly the same performance as Parquet for many similar workloads
Simplified data management model
Can handle a new class of streaming use cases and workloads
Great… let’s just use Kudu from now on:
● We can ingest data with great write throughput
● Support analytic queries
● Support random access writes
What’s not to love?
Ship It!
10

There’s a catch...
… it’s your data model
Good news! If you have figured this out with HDFS and Parquet, you’re not too far off.
Things to consider:
● Access pattern and partition scheme (similar to partitioning data parquet)
○ Has a big role to play in parallelism of your queries
● Cardinality of your attributes
○ Affects what type of column encoding you decide to use
● Key structure
○ You get only one, use it wisely
More on this can be found at : http://getkudu.io/docs/schema_design.html
11

Let’s put it all together
I have a fruit stand
I sell my fruits via phone order to remote
buyers
My transactions look something like:
Orders(orderID,orderTS,fruit,price,customerID,
customerPhone,customerAddress)
12

Impala DDL for Kudu
CREATE EXTERNAL TABLE `strata_fruits_expanded` (
`_ts` BIGINT,
`_id` STRING,
`fruit` STRING,
`country_code` STRING,
`country_area_code` STRING,
`phone_num` STRING,
`message_date` BIGINT,
`price` FLOAT,
`keyword` STRING
)
DISTRIBUTE BY HASH (_ts) INTO 60 BUCKETS
TBLPROPERTIES(
'storage_handler' =
'com.cloudera.kudu.hive.KuduStorageHandler',
'kudu.table_name' = 'strata_fruits_expanded',
'kudu.master_addresses' = '10.xxx.xxx.xxx:7051',
'kudu.key_columns' = '_ts, _id'
);
Key
Key
13

Impala DDL for Kudu
`_ts` BIGINT,
`_id` STRING,
`fruit` STRING,
`phone_num` STRING,
`price` FLOAT,
`keyword` STRING
)
TBLPROPERTIES(
'storage_handler' =
);
attributes
attributes
Low cardinality attributes -- things I
want to group by -- are great
candidates for dictionary encoding
14

Impala DDL for Kudu
`_ts` BIGINT,
`_id` STRING,
`fruit` STRING,
`phone_num` STRING,
`price` FLOAT,
`keyword` STRING
)
TBLPROPERTIES(
'storage_handler' =
);
Partition Scheme
How you distribute your data directly
impacts your ability to process in
parallel as well as any predicate
push-down type of operations Kudu
can perform
For large tables, such as fact tables,
aim for as many tablets as you have
cores in the cluster -- but figure out
what else you are running as well.
15

Let’s see it in action….
16
Kafka Source Topic
Spark Streaming
App
Data Writer Kudu
ZoomdataKafka Sink Topic
write read
Streaming Source

Let’s see it in action… not actually that simple
17
Kafka Source Topic
Spark Streaming
App
Data Writer API
Writer Client
Data Writer
Writer Server
Register
Read request
Kudu, Solr,
Elastic, etc...
ZoomdataKafka Sink Topic
write read
Streaming Source

Special Thanks
Anton Gorshkov: For his original streaming with kafka fruit stand demo
The Cloudera Kudu Team: Specifically Todd Lipcon for all the insight
into Kudu optimization
Nexmo: For use of their SaaS SMS service in this demo
18

Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East talk by Ruhollah Farchtchi

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East talk by Ruhollah Farchtchi (20)

More from Spark Summit (20)

Recently uploaded (20)

Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East talk by Ruhollah Farchtchi