SlideShare a Scribd company logo

Real-time Analytics with Apache Flink and Druid

J
Jan Graßegger

This document discusses using Apache Flink and Druid for real-time analytics. It describes Druid as an online analytical processing (OLAP) system that is column-oriented, distributed, and uses built-in data sharding based on time windows. It also introduces Tranquility, which helps ingest real-time data into Druid from systems like Kafka, Spark, and Flink. The document proposes a processing architecture using Kafka, Flink, Druid and Tranquility, with HDFS for replays, to enable real-time reporting with capabilities for replays from HDFS and Kafka.

Engineering
1 of 18
Downloaded 46 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
REAL-TIME ANALYTICS WITH APACHE FLINK AND DRUID Berlin Buzzwords 2016 Jan Graßegger - @gesundkrank
DATA ENGINEER @
OUR DATA 70,000EVENTS PER SECOND 50DIMENSIONS 20METRICS
DRUID
DRUID ‣ Online Analytical Processing (OLAP) System ‣ Column-oriented ‣ Distributed ‣ Built-in data sharding based on time windows ‣ JSON query language
DATA STRUCTURES Column TOP PRIVATE DOMAIN battle.net battle.net noxxic.com noxxic.com Strings to Integers battle.net 5 noxxic.com 6 Encoded column data [5, 5, 6, 6]
DATA STRUCTURES Column Bitmap Indices battle.net [1, 1, 0, 0] noxxic.com [0, 0, 1, 1] TOP PRIVATE DOMAIN battle.net battle.net noxxic.com noxxic.com
FIREHOSES FIREHOSES
APACHE FLINK
PROCESSING ?Kafka Flink Druid
TRANQUILITY
TRANQUILITY ‣ Helps ingesting real-time data into Druid ‣ Provides adapters for Samza, Spark, Storm and Flink ‣ Standalone HTTP and Kafka applications
Kafka Flink Druid Tranquility PROCESSING Replays?
LAMBDA
KAPPA
Kafka Flink Druid Tranquility HDFS for replays PROCESSING
RESULTS ▸Kappa-like architecture that’s able to do replays from HDFS & Kafka ▸Added Flink sink to Tranquility ▸“Hacked“ replays into Tranquility ▸Real-Time Reporting
QUESTIONS?

More Related Content

What's hot (20)

PDF
Intro to Delta Lake
Databricks
 
PDF
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
PDF
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
PPTX
Databricks Fundamentals
Dalibor Wijas
 
PPTX
Delta lake and the delta architecture
Adam Doyle
 
PPTX
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
spark-project
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
PDF
adb.pdf
AdityaMehta724216
 
PDF
3D: DBT using Databricks and Delta
Databricks
 
PPTX
Free Training: How to Build a Lakehouse
Databricks
 
PDF
Getting Started with Delta Lake on Databricks
Knoldus Inc.
 
PDF
GCP for Apache Kafka® Users: Stream Ingestion and Processing
confluent
 
PDF
Databricks Delta Lake and Its Benefits
Databricks
 
PDF
Building End-to-End Delta Pipelines on GCP
Databricks
 
PDF
Building Robust ETL Pipelines with Apache Spark
Databricks
 
PDF
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
 
PDF
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Paris Data Engineers !
 
PDF
Time to Talk about Data Mesh
LibbySchulze
 
PPTX
Apache NiFi in the Hadoop Ecosystem
DataWorks Summit/Hadoop Summit
 
PDF
Intro to HBase
alexbaranau
 
Intro to Delta Lake
Databricks
 
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
Databricks Fundamentals
Dalibor Wijas
 
Delta lake and the delta architecture
Adam Doyle
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
spark-project
 
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
adb.pdf
AdityaMehta724216
 
3D: DBT using Databricks and Delta
Databricks
 
Free Training: How to Build a Lakehouse
Databricks
 
Getting Started with Delta Lake on Databricks
Knoldus Inc.
 
GCP for Apache Kafka® Users: Stream Ingestion and Processing
confluent
 
Databricks Delta Lake and Its Benefits
Databricks
 
Building End-to-End Delta Pipelines on GCP
Databricks
 
Building Robust ETL Pipelines with Apache Spark
Databricks
 
Apache Iceberg - A Table Format for Hige Analytic Datasets
Alluxio, Inc.
 
Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard
Paris Data Engineers !
 
Time to Talk about Data Mesh
LibbySchulze
 
Apache NiFi in the Hadoop Ecosystem
DataWorks Summit/Hadoop Summit
 
Intro to HBase
alexbaranau
 

Viewers also liked (20)

PDF
Aggregated queries with Druid on terrabytes and petabytes of data
Rostislav Pashuto
 
PPTX
Scalable Real-time analytics using Druid
DataWorks Summit/Hadoop Summit
 
PDF
Real-time analytics with Druid at Appsflyer
Michael Spector
 
PPT
Case Study: Realtime Analytics with Druid
Salil Kalia
 
PDF
Interactive analytics at scale with druid
Julien Lavigne du Cadet
 
PDF
Data Analytics with Druid
Yousun Jeong
 
PPTX
Druid realtime indexing
Seoeun Park
 
PPTX
Druid at Hadoop Ecosystem
Slim Bouguerra
 
PPTX
Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid
Tony Ng
 
PPTX
Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...
SoftServe
 
PPTX
Click-Through Example for Flink’s KafkaConsumer Checkpointing
Robert Metzger
 
PPTX
Apache Kylin - OLAP Cubes for SQL on Hadoop
Ted Dunning
 
PDF
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Xu Jiang
 
PPTX
Realtime streaming architecture in INFINARIO
Jozo Kovac
 
PDF
Apache Provisionr (incubating) - Bucharest JUG 10
Andrei Savu
 
PDF
Helio, a Continues Real-Time Fraud Detection and Monitoring Solution
Amir Sedighi
 
PDF
Strata lightening-talk
Danny Yuan
 
PPTX
Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...
Data Con LA
 
PDF
Druid @ branch
Biswajit Das
 
PPTX
Apache Kylin Streaming
hongbin ma
 
Aggregated queries with Druid on terrabytes and petabytes of data
Rostislav Pashuto
 
Scalable Real-time analytics using Druid
DataWorks Summit/Hadoop Summit
 
Real-time analytics with Druid at Appsflyer
Michael Spector
 
Case Study: Realtime Analytics with Druid
Salil Kalia
 
Interactive analytics at scale with druid
Julien Lavigne du Cadet
 
Data Analytics with Druid
Yousun Jeong
 
Druid realtime indexing
Seoeun Park
 
Druid at Hadoop Ecosystem
Slim Bouguerra
 
Pulsar: Real-time Analytics at Scale with Kafka, Kylin and Druid
Tony Ng
 
Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziye...
SoftServe
 
Click-Through Example for Flink’s KafkaConsumer Checkpointing
Robert Metzger
 
Apache Kylin - OLAP Cubes for SQL on Hadoop
Ted Dunning
 
Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive
Xu Jiang
 
Realtime streaming architecture in INFINARIO
Jozo Kovac
 
Apache Provisionr (incubating) - Bucharest JUG 10
Andrei Savu
 
Helio, a Continues Real-Time Fraud Detection and Monitoring Solution
Amir Sedighi
 
Strata lightening-talk
Danny Yuan
 
Big Data Day LA 2016/ Big Data Track - Real Time Analytics with Druid - Guill...
Data Con LA
 
Druid @ branch
Biswajit Das
 
Apache Kylin Streaming
hongbin ma
 
Ad

Similar to Real-time Analytics with Apache Flink and Druid (20)

PDF
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
HostedbyConfluent
 
PDF
How To Use Kafka and Druid to Tame Your Router Data (Rachel Pedreschi, Imply ...
confluent
 
PDF
How To Use Kafka and Druid to Tame Your Router Data (Rachel Pedreschi and Eri...
confluent
 
PDF
Apache Druid 101
Data Con LA
 
PPTX
Realtime classroom analytics powered by apache druid
Karthik Deivasigamani
 
PDF
Streaming sql and druid
arupmalakar
 
PDF
Druid
Dori Waldman
 
PPTX
Scalable olap with druid
Kashif Khan
 
PPTX
January 2016 Flink Community Update & Roadmap 2016
Robert Metzger
 
PDF
Self Service Analytics at Twitch
Imply
 
PPTX
Flink Streaming @BudapestData
Gyula Fóra
 
PPTX
Realtime data processing with Flink and Druid by Youngpyo Lee, SKT
Metatron
 
PDF
Flink Forward SF 2017: Cliff Resnick & Seth Wiesman - From Zero to Streami...
Flink Forward
 
PDF
Stream processing with Apache Flink @ OfferUp
Bowen Li
 
PPTX
Workshop híbrido: Stream Processing con Flink
confluent
 
PPTX
The of Operational Analytics Data Store
Rommel Garcia
 
PDF
Five Fabulous Sinks for Your Kafka Data. #3 will surprise you! (Rachel Pedres...
confluent
 
PPTX
Real time analytics
Leandro Totino Pereira
 
PDF
Druid at Strata Conf NY 2016.pdf
HimanshuGupta936
 
PDF
Fast analytics kudu to druid
Worapol Alex Pongpech, PhD
 
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
HostedbyConfluent
 
How To Use Kafka and Druid to Tame Your Router Data (Rachel Pedreschi, Imply ...
confluent
 
How To Use Kafka and Druid to Tame Your Router Data (Rachel Pedreschi and Eri...
confluent
 
Apache Druid 101
Data Con LA
 
Realtime classroom analytics powered by apache druid
Karthik Deivasigamani
 
Streaming sql and druid
arupmalakar
 
Druid
Dori Waldman
 
Scalable olap with druid
Kashif Khan
 
January 2016 Flink Community Update & Roadmap 2016
Robert Metzger
 
Self Service Analytics at Twitch
Imply
 
Flink Streaming @BudapestData
Gyula Fóra
 
Realtime data processing with Flink and Druid by Youngpyo Lee, SKT
Metatron
 
Flink Forward SF 2017: Cliff Resnick & Seth Wiesman - From Zero to Streami...
Flink Forward
 
Stream processing with Apache Flink @ OfferUp
Bowen Li
 
Workshop híbrido: Stream Processing con Flink
confluent
 
The of Operational Analytics Data Store
Rommel Garcia
 
Five Fabulous Sinks for Your Kafka Data. #3 will surprise you! (Rachel Pedres...
confluent
 
Real time analytics
Leandro Totino Pereira
 
Druid at Strata Conf NY 2016.pdf
HimanshuGupta936
 
Fast analytics kudu to druid
Worapol Alex Pongpech, PhD
 
Ad

Recently uploaded (20)

PDF
Electrical Engineer operation Supervisor
ssaruntatapower143
 
PPTX
MODULE 04 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 
PPTX
2025 CGI Congres - Surviving agile v05.pptx
Derk-Jan de Grood
 
PDF
mbse_An_Introduction_to_Arcadia_20150115.pdf
henriqueltorres1
 
PPT
New_school_Engineering_presentation_011707.ppt
VinayKumar304579
 
PDF
AI TECHNIQUES FOR IDENTIFYING ALTERATIONS IN THE HUMAN GUT MICROBIOME IN MULT...
vidyalalltv1
 
PPTX
美国电子版毕业证南卡罗莱纳大学上州分校水印成绩单USC学费发票定做学位证书编号怎么查
Taqyea
 
PPTX
What is Shot Peening | Shot Peening is a Surface Treatment Process
Vibra Finish
 
PDF
3rd International Conference on Machine Learning and IoT (MLIoT 2025)
ClaraZara1
 
PPTX
OCS353 DATA SCIENCE FUNDAMENTALS- Unit 1 Introduction to Data Science
A R SIVANESH M.E., (Ph.D)
 
PPTX
Water Resources Engineering (CVE 728)--Slide 3.pptx
mohammedado3
 
PDF
methodology-driven-mbse-murphy-july-hsv-huntsville6680038572db67488e78ff00003...
henriqueltorres1
 
PPTX
Final Major project a b c d e f g h i j k l m
bharathpsnab
 
PDF
Electrical Machines and Their Protection.pdf
Nabajyoti Banik
 
PDF
methodology-driven-mbse-murphy-july-hsv-huntsville6680038572db67488e78ff00003...
henriqueltorres1
 
PDF
Data structures notes for unit 2 in computer science.pdf
sshubhamsingh265
 
PPTX
Biosensors, BioDevices, Biomediccal.pptx
AsimovRiyaz
 
PDF
AN EMPIRICAL STUDY ON THE USAGE OF SOCIAL MEDIA IN GERMAN B2C-ONLINE STORES
ijait
 
PDF
Viol_Alessandro_Presentazione_prelaurea.pdf
dsecqyvhbowrzxshhf
 
PPTX
Lecture 1 Shell and Tube Heat exchanger-1.pptx
mailforillegalwork
 
Electrical Engineer operation Supervisor
ssaruntatapower143
 
MODULE 04 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 
2025 CGI Congres - Surviving agile v05.pptx
Derk-Jan de Grood
 
mbse_An_Introduction_to_Arcadia_20150115.pdf
henriqueltorres1
 
New_school_Engineering_presentation_011707.ppt
VinayKumar304579
 
AI TECHNIQUES FOR IDENTIFYING ALTERATIONS IN THE HUMAN GUT MICROBIOME IN MULT...
vidyalalltv1
 
美国电子版毕业证南卡罗莱纳大学上州分校水印成绩单USC学费发票定做学位证书编号怎么查
Taqyea
 
What is Shot Peening | Shot Peening is a Surface Treatment Process
Vibra Finish
 
3rd International Conference on Machine Learning and IoT (MLIoT 2025)
ClaraZara1
 
OCS353 DATA SCIENCE FUNDAMENTALS- Unit 1 Introduction to Data Science
A R SIVANESH M.E., (Ph.D)
 
Water Resources Engineering (CVE 728)--Slide 3.pptx
mohammedado3
 
methodology-driven-mbse-murphy-july-hsv-huntsville6680038572db67488e78ff00003...
henriqueltorres1
 
Final Major project a b c d e f g h i j k l m
bharathpsnab
 
Electrical Machines and Their Protection.pdf
Nabajyoti Banik
 
methodology-driven-mbse-murphy-july-hsv-huntsville6680038572db67488e78ff00003...
henriqueltorres1
 
Data structures notes for unit 2 in computer science.pdf
sshubhamsingh265
 
Biosensors, BioDevices, Biomediccal.pptx
AsimovRiyaz
 
AN EMPIRICAL STUDY ON THE USAGE OF SOCIAL MEDIA IN GERMAN B2C-ONLINE STORES
ijait
 
Viol_Alessandro_Presentazione_prelaurea.pdf
dsecqyvhbowrzxshhf
 
Lecture 1 Shell and Tube Heat exchanger-1.pptx
mailforillegalwork
 

Real-time Analytics with Apache Flink and Druid