Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)

@KaiWaehner - www.kai-waehner.de - Data in Motion for the Industrial IoT
as Data Hub for
Crypto, NFT, Metaverse
(Beyond the Buzz!)
kai-waehner.de | @KaiWaehner | Field CTO @ Confluent

kai-waehner.de | @KaiWaehner | Data Streaming as Data Hub for Crypto, Blockchain, and Metaverse
1. When (not) to use Blockchain and Crypto
2. Data Streaming for Real-time Analytics at Scale
3. Examples for Data Streaming with Crypto Data
4. Real-World Projects
5. Choosing the Right Tool for the Job
Agenda

Potential impact
of cryptocurrency
on financial
services
6
https://www.hedgethink.com/the-cryptocurrency-evolution/potential-impact-of-cryptocurrency-on-financial-services/

Bitcoin – The Tip of the Iceberg!
7
‘Bitcoin’ compared to
‘Blockchain’, ‘Crypto’, ‘Ethereum'

Web3 beyond the hype: 5 types of digital assets
8
Native tokens, which are the monetary incentives used to compensate nodes for
maintaining and updating the respective blockchain
Stablecoins, which are supposed to represent cash on the blockchain and are pegged to
fiat currencies like the US dollar, or central bank digital currencies (CBDCs), which are
regulated by a central bank 2
Governance tokens, which are tokens that represent voting rights on the functional
parameters of smart contracts
Non-fungible tokens (NFTs), which are a unique, indivisible digital asset with provable
ownership
Digital assets that represent claims on real-world assets such as commodities, real
estate, or intellectual property, and are “tokenized” into divisible digital assets on the
blockchain
https://www.mckinsey.com/industries/financial-services/our-insights/web3-beyond-the-hype

Blockchain vs. Distributed Ledger Technology (DLT)
9
https://www.da.ventures/post/gaining-clarity-on-key-terminology-bitcoin-versus-blockchain-versus-distributed-ledger-technology

The Cryptocurrency Trilemma
10
SCALABILITY
SECURITY DECENTRALIZATION
Pick A, B, or C
A B
C

Challenges and Concerns of Blockchains
• Technical and organizational
complexity
• Transaction speed and
scalability
• Energy consumption
• Security
• Data tenancy
• Lifecycle costs
• Ease of integration

Purpose-built solutions
• Ripple: Payment settling, currency exchange and remittance system
• Diem (former Libra): Global payment system and financial infrastructure
• IOTA: Internet of Things transactions
Non-Blockchain DLTs
to solve the challenges to provide better performance and scalability
12
Open and flexible solution for many use cases
• Apache Kafka: Open, scalable, real time Data Streaming
* dead

STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C C
C
Data Streaming

Data Streaming in the Finance Industry
Check past Kafka Summit videos for details about the use cases:
https://kafka-summit.org/past-events/

Tamper-Proof Data Streaming for
(some of the) Use Cases in the Finance Industry
Real Time Processing Digital Transformation Strategic Goals
Short-Sale Risk
Calculation / Trade
Approval
Mainframe Offloading
and Replacement
Instant Payment
Next-Best
Offer
Robot Process Automation
(e.g. Know Your Customer, KYC)
Customer Service
(e.g. Chat Bots)
IT
Modernization
Regulatory
Reporting
Fraud Detection
Anomaly Detection
Across Assets and Locations
Derivatives
Pricing Compliance
Trading Post-
Processing
Strategic
Planning and
Simulations

Apache Kafka – The de facto standard for Data Streaming
is the Underpinning of an Event-driven Architecture
Microservices
Mainframes
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Stream processing apps
Connectors
Connectors
Stream processing apps

Kafka is a cloud-native data streaming platform!
à More than just data ingestion or message queue
DWH
APP
STREAM
PROCESSING
CONNECTORS
ksqlDB
KStreams
APP
Streaming ETL
Data Processing
Real-time Analytics
Stateless and Stateful
Business Applications
Fully-managed
Pipelines
Connectivity to
Data Infrastructure,
SaaS, AI/ML
Data Governance
Connectivity
Filtering and Routing
Change Data Capture
Built-in Scale and Fault Tolerance
Oracle
DB
ORACLE CDC
SOURCE
PREMIUM
CONNECTOR
Real-time Data Sharing
across Hybrid and Multi-Cloud
Storage
Backpressure Handling
Slow Consumers
Replayability

Data Streaming across the Globe
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Warehouses, Data Lakes, Apps, APIs, SaaS
Aggregation of Edge
Deployments with
Replication (Aggregation)
Disaster Recovery
Operations with
Multi-Region Clusters
for RPO=0 and RTO~0
Global Data Streaming with
Replication and Cluster Linking

Real-time
High Throughput
Decentralized database
Distributed log of records
Immutable log
Replication
High availability
Decoupling of applications / clients
Role-based access control to data
Tamper-Proof
Encrypted payloads
Deployment across independent organizations
is not a Blockchain!
X

Kafka AND Blockchain
30
Kafka
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)
Blockchain
(Bitcoin)
Blockchain
(Ethereum)
Kafka-native
Blockchain or
Distributed Ledger
Kafka AS Blockchain
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)

Trading /
Payments
for Transactional
Banking Workloads
3rd Party Crypto Data
as Foundation of the Crypto Application
Application Logs
for Analytical Workloads
Data Producers for Crypto Cybersecurity

Streams
logs-customer-trading
Transaction Alerts for
Situational Awareness
Machine Learning
Predictions via UDFs
Data Aggregation
suspicious-events
aggregation-analytics
Real-Time Data Processing for Crypto Threat Intelligence
Curated Big Data for
Data Science Teams

Compliance
Threat Detection
Near Real-time
Data Consumers for Alerting and Regulatory Reporting
No constraints on integration flows
Data curation on the fly
Flexible choice of (multiple) consumers
Business Intelligence
Analytical Workloads
Batch
Alerting App
Transactional Workloads
Real-time

Connect
REST API
Connect
logs-customer-trading
suspicious-events
fraud-alert-events
Streams
Data
Correlation
Anomaly
Detection
Kafka as Data Hub for Real-Time Cyber Analytics in the Crypto World
Python
Fraud
Alerting
Streams

Metaverse: Live commerce with real-time data correlation
Online and offline shopping via social commerce
Including integration of CRM, loyalty, inventory, chatbots, location-based services, augmented reality, etc.
35

Data Streaming as Data Hub for Crypto, Blockchain, and Metaverse
Interact with other people in the metaverse.
Upsell rare items for your games.
Provide context-specific pricing.
Integrate with crypto wallets and NFT markets.
All automated. In real-time. At scale.
Data Consolidation
Streaming ETL
(Kafka Connect + Kafka Streams)
Mobile App
+ AR Glasses
(Swift on iOS)
(4) Receive
NFT recommendation
and context-specific price
Crypto
Payment Service
Buy Now Pay Later (BNPL)
(Ethereum API)
Chatbot
External Speech NLP
(REST API)
(1) Interest in
rare game item
(5) Order and pay
NFT for game item
(2) Live Q&A with chatbot
(6) Execute Crypto
transaction and confirm NFT
CRM / Loyalty
Salesforce
(Kafka Connect)
NFT Marketplace
Live monitoring and
item bidding process
(ksqlDB + OpenSea API)
(3) Context-specific
pricing engine

Kafka
AS
Blockchain
40

R3 Corda: Kafka AS Blockchain / DLT
41
https://medium.com/corda/corda-5-the-road-ahead-part-3-introduction-to-the-architecture-8c43bb19ac81
Rearchitected: V5 migrates to Kafka
High availability for critical services + cost-effective way to scale
(horizontally) to deal with ‘burst-y’ and high-volume throughputs
Fully redundant, worker-based architecture
Kafka cluster as the message broker to facilitate communication between
node services
Compacted topics (logs) as the mechanism to store and retrieve most
recent states
Key components: Process engine, database integration, Crypto
integration, RPC service (HTTP), Monitoring

Chainlink
Industry standard oracle network for connecting smart contracts to the real world
Transitioning from traditional time series-based monitoring toward an event-driven architecture and alerting approach
https://jobs.lever.co/chainlink/b0094754-6d1e-4c55-8dbc-622bc4d8a5ab

Kafka
IN
Blockchain Tools
43

TokenAnalyst
44
https://www.confluent.io/en-gb/blog/reliable-fast-access-to-on-chain-data-insights/
Integrate blockchain data with its analytics tools.
Kafka Streams provides a stateful streaming application to prevent using invalid blocks in downstream aggregate calculations.
Kafka Connect for the integration with databases and data lake.

EthVM – Blockchain Explorer
45
https://github.com/EthVM/EthVM
https://www.bitcoininsider.org/article/66671/ethvm-first-open-source-block-explorer-powered-kafka
Open-source Ethereum Blockchain Data
Processing and Analytics Engine with a
client-side Block Explorer
Tool for blockchain auditing and decision
Verify the execution of transactions and
smart contracts, check balances, and
monitor gas prices
Built with Kafka Connect, Kafka Streams,
Schema Registry

kaleido – Rest API for Crypto
46
https://www.kaleido.io/blockchain-platform/rest-api-gateway
Enterprise grade blockchain APIs to
deploy and manage Smart Contracts,
send Ethereum transactions, and query
blockchain data
Hides the complexities of Ethereum
transaction submission, thick Web3
client libraries, nonce management, RLP
encoding, transaction signing, and
smart contract management.
REST APIs for your on-chain logic &
data
Backed by a fully-managed high
throughput Apache Kafka
infrastructure

Kafka
AND
Blockchain
47

Kafka AND Blockchain:
Financial services platform
combining Kafka and Blockchains
• Provides the speed and convenience of traditional
exchanges and the security of non-custodial
approaches.
• Invest in, make payments with and trade Bitcoin,
Ethereum, NEO, and other digital assets.
• The exchange is the first of its kind, offering non-
custodial cross-chain trading with the full power of a
real order book
• Deterministic replayability in its exact order at any time
48
https://community.nash.io/t/first-of-its-kind-non-custodial-exchange-using-confluent-cloud/8254

CUSTODIGIT – A Platform for Digital Assets
For cryptocurrencies (bitcoins, etc.) and digital assets
Secure storage of wallets
Sending and receiving on the blockchain
Trading via brokers and exchanges
Regulated environment
Kafka as central nervous system
Workflow orchestration with distributed saga design pattern
https://events.confluent.io/datainmotiontour20211

Kafka vs. Blockchain?
51
Use Kafka and
Blockchain together to
combine the benefits of
both
(where this makes
sense!)

Kafka vs. Blockchain?
52
Use Apache Kafka for
• Enterprise infrastructure
• Open, scalable, real-time
requirements
• Flexible architectures for many
use cases
Use Blockchain for
• Deployment over various independent
organizations
• Participants verify the distributed ledger contents
themselves.
• Specific use cases
• Server-side managed and controlled by
multiple organizations
• Scenarios where the business value overturns
the added complexity and project risk

Is ‘tamper-proof’
all you need?
53

54
Apache Kafka is an immutable append-only commit log

KafkaBlockchain – A library for tamper-proof Kafka streaming
https://github.com/ai-coin/KafkaBlockchain

https://github.com/ai-coin/KafkaBlockchain
KafkaBlockchain – A library for tamper-proof Kafka streaming

What about long-term storage in Kafka?
57

Today, Kafka works well
for recent events, short
horizon storage, and
manual data balancing
Kafka’s present-day design offers
extraordinarily low messaging latency by
storing topic data on fast disks that are
collocated with brokers. This is usually good.
But sometimes, you need to store a huge
amount of data for a long time.
Blockchain is such a use case!
Kafka
Processing
App
Storage
Transactions, auth,
quota
enforcement,
compaction, ...

Confluent Tiered Storage for Kafka
Object Store
Processing Storage
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Kafka
Apps
Store Forever (Tamper-Proof)
Older data is offloaded to inexpensive object storage,
permitting it to be consumed at any time. Using
KafkaBlockchain, storage can be made tamper-proof and
immutable
Save $$$
Storage limitations, like capacity and duration, are effectively
uncapped.
Instantaneously scale up and down
Your Kafka clusters will be able to automatically self-balance
load and hence elastically scale

Secure, tamper-proof, encrypted off-chain data streaming
Time
P
C1
C2
Payment transaction
Real-time data processing and analytics of historical events with one scalable infrastructure
Compliance check on historical data
Real-time
off-chain
monitoring

@KaiWaehner - www.kai-waehner.de - Data in Motion for the Industrial IoT
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.

Kai Waehner
Field CTO
kai.waehner@confluent.io
confluent.io
kai-waehner.de
@KaiWaehner
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)

Recommended

More Related Content

What's hot (20)

Similar to Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!) (20)

More from Kai Wähner (20)

Recently uploaded (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)