Change Data Capture Pipelines with Debezium and Kafka Streams (Gunnar Morling, Red Hat) Kafka Summit 2020

Change Data Capture Pipelines With
Debezium and Kafka Streams
Gunnar Morling
Software Engineer
@gunnarmorling

Recap: Debezium
What's Change Data Capture?
Use Cases
Kafka Streams with Quarkus
Supersonic Subatomic Java
The Kafka Streams Extension
Debezium + Kafka Streams =
Data Enrichment
Auditing
Aggregate View Materialisation
1
2
3

Gunnar Morling
Open source software engineer at Red Hat
Debezium
Quarkus
Hibernate
Spec Lead for Bean Validation 2.0
Other projects: Deptective, MapStruct
Java Champion
#Debezium @gunnarmorling

@gunnarmorling
Postgres
MySQL
Kafka Connect Kafka ConnectApache Kafka
DBZ PG
DBZ
MySQL
Search Index
ES
Connector
JDBC
Connector
ES
Connector
ISPN
Connector
Cache
Debezium
Enabling Zero-Code Data Streaming Pipelines
Data
Warehouse
#Debezium

@gunnarmorling
Debezium
Low-Latency Change Data Streaming
https://medium.com/convoy-tech/
#Debezium

@gunnarmorling
CDC – "Liberation for Your Data"
#Debezium

{
"before": null,
"after": {
"id": 1004,
"first_name": "Anne",
"last_name": "Kretchmar",
"email": "annek@noanswer.org"
},
"source": {
"name": "dbserver1",
"server_id": 0,
"ts_sec": 0,
"file": "mysql-bin.000003",
"pos": 154,
"row": 0,
"snapshot": true,
"db": "inventory",
"table": "customers"
},
"op": "c",
"ts_ms": 1486500577691
}
Change Event Structure
Key: Primary key of table
Value: Describing the change event
Old row state
New row state
Metadata
@gunnarmorling#Debezium

Recap: Debezium
Use Cases
1
2
3
Data Enrichment
Auditing

@gunnarmorling
Quarkus
“ A Kubernetes Native Java stack tailored for
OpenJDK HotSpot and GraalVM, crafted from the
best of breed Java libraries and standards.
#Debezium

Quarkus
Developer joy
Imperative and Reactive
Best-of-breed libraries

Quarkus
Management of topology
Health checks
Dev Mode
Support for native binaries via GraalVM

Recap: Debezium
Use Cases
1
3
2
Data Enrichment
Auditing

@gunnarmorling
Auditing
Source DB Kafka Connect Apache Kafka
DBZ
Customer Events
CRM
Service
#Debezium

@gunnarmorling
Auditing
DBZ
Customer Events
CRM
Service
Id User Use Case
tx-1 Bob Create Customer
tx-2 Sarah Delete Customer
tx-3 Rebecca Update Customer
"Transactions" table
#Debezium

@gunnarmorling
Auditing
DBZ
Customer Events
Transactions
CRM
Service
Id User Use Case
#Debezium

@gunnarmorling
Auditing
DBZ
Customer Events
Transactions
CRM
Service
Kafka
Streams
Id User Use Case
#Debezium

@gunnarmorling
Auditing
DBZ
Customer Events
Transactions
CRM
Service
Kafka
Streams
Id User Use Case
Enriched Customer Events
#Debezium

@gunnarmorling
Auditing
{
"before": {
"id": 1004,
"email": "annek@example.com"
},
"after": {
"id": 1004,
},
"source": {
"table": "customers",
"txId": "tx-3"
},
"op": "u",
"ts_ms": 1486500577691
}
Customers
#Debezium

@gunnarmorling
{
"before": {
"id": 1004,
},
"after": {
"id": 1004,
},
"source": {
"txId": "tx-3"
},
"op": "u",
"ts_ms": 1486500577691
}
{
"before": null,
"after": {
"id": "tx-3",
"user": "Rebecca",
"use_case": "Update customer"
},
"source": {
"table": "transactions",
"txId": "tx-3"
},
"op": "c",
"ts_ms": 1486500577691
}
Transactions Customers
{
"id": "tx-3"
}
#Debezium

{
"id": "tx-3"
}
{
"before": {
"id": 1004,
},
"after": {
"id": 1004,
},
"source": {
"txId": "tx-3"
},
"op": "u",
"ts_ms": 1486500577691
}
Transactions Customers
@gunnarmorling
{
"before": null,
"after": {
"id": "tx-3",
"user": "Rebecca",
},
"source": {
"table": "transactions",
"txId": "tx-3"
},
"op": "c",
"ts_ms": 1486500577691
}
#Debezium

@gunnarmorling
{
"before": {
"id": 1004,
},
"after": {
"id": 1004,
},
"source": {
"txId": "tx-3",
"user": "Rebecca",
},
"op": "u",
"ts_ms": 1486500577691
}
Enriched Customers
Auditing
#Debezium

@gunnarmorling
@Override
public KeyValue<JsonObject, JsonObject>
transform(JsonObject key, JsonObject value) {
boolean enrichedAllBufferedEvents =
enrichAndEmitBufferedEvents();
if (!enrichedAllBufferedEvents) {
bufferChangeEvent(key, value);
return null;
}
KeyValue<JsonObject, JsonObject> enriched =
enrichWithTxMetaData(key, value);
if (enriched == null) {
bufferChangeEvent(key, value);
}
return enriched;
}
Auditing
Non-trivial join implementation
no ordering across topics
need to buffer change events
until TX data available
bit.ly/debezium-auditlogs
#Debezium

Aggregate View Materialization
From Multiple Topics to One View
PurchaseOrder
OrderLine
{
"purchaseOrderId" : "order-123",
"orderDate" : "2020-08-24",
"customerId": "customer-123",
"orderLines" : [
{
"orderLineId" : "orderLine-456",
"productId" : "product-789",
"quantity" : 2,
"price" : 59.99
},
{
"orderLineId" : "orderLine-234",
"productId" : "product-567",
"quantity" : 1,
"price" : 14.99
}
1
n

Non-Key Joins (KIP-213)
KTable<Long, OrderLine> orderLines = ...;
KTable<Integer, PurchaseOrder> purchaseOrders = ...;
KTable<Integer, PurchaseOrderWithLines> purchaseOrdersWithOrderLines = orderLines
.join(
purchaseOrders,
orderLine -> orderLine.purchaseOrderId,
(orderLine, purchaseOrder) -> new OrderLineAndPurchaseOrder(orderLine, purchaseOrder))
.groupBy(
(orderLineId, lineAndOrder) -> KeyValue.pair(lineAndOrder.purchaseOrder.id, lineAndOrder))
.aggregate(
PurchaseOrderWithLines::new,
(Integer key, OrderLineAndPurchaseOrder value, PurchaseOrderWithLines agg) -> agg.addLine(value),
(Integer key, OrderLineAndPurchaseOrder value, PurchaseOrderWithLines agg) -> agg.removeLine(value)
);

Awareness of Transaction Boundaries
TX metadata in change events (e.g. dbserver1.inventory.orderline)
{
"before": null,
"after": { ... },
"source": { ... },
"op": "c",
"ts_ms": "1580390884335",
"transaction": {
"id": "571",
"total_order": "1",
"data_collection_order": "1"
}
}

Awareness of Transaction Boundaries
Topic with BEGIN/END markers
Enable consumers to buffer all events
of one transaction
@gunnarmorling
{
"transactionId" : "571",
"eventType" : "begin transaction",
"ts_ms": 1486500577125
}
{
"transactionId" : "571",
"ts_ms": 1486500577691,
"eventType" : "end transaction",
"eventCount" : [
{
"name" : "dbserver1.inventory.order",
"count" : 1
},
{
"name" : "dbserver1.inventory.orderLine",
"count" : 5
}
]
}
BEGIN END
#Debezium

Takeaways
Many Use Cases for Debezium and Kafka Streams
Data enrichment
Creating aggregated events
Interactive query services for legacy databases

Takeaways
Many Use Cases for Debezium and Kafka Streams
Data enrichment
Creating aggregated events
Interactive query services for legacy databases
Debezium + Kafka
Streams =

Resources
Website:
Examples:
Latest news: @debezium
debezium.io
github.com/debezium/debezium-examples/

gunnar@hibernate.org
@gunnarmorling
@gunnarmorling
Q&A
#Debezium

Change Data Capture Pipelines with Debezium and Kafka Streams (Gunnar Morling, Red Hat) Kafka Summit 2020

Change Data Capture Pipelines with Debezium and Kafka Streams (Gunnar Morling, Red Hat) Kafka Summit 2020

Recommended

More Related Content

What's hot (20)

Similar to Change Data Capture Pipelines with Debezium and Kafka Streams (Gunnar Morling, Red Hat) Kafka Summit 2020 (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

Change Data Capture Pipelines with Debezium and Kafka Streams (Gunnar Morling, Red Hat) Kafka Summit 2020