PartnerSkillUp_Enable a Streaming CDC Solution

Partner Skill Up:
Enable a Streaming CDC Solution
Tim Spann
Principal Developer Advocate in Data In Motion for Cloudera, Global
tspann@cloudera.com
Salvador Almazan
Partner Solutions Engineer, US
salmazan@cloudera.com

2
© Cloudera, Inc. All rights reserved.
Today’s Leads
CLOUDERA TEAM
Meet the Cloudera team
Salvador Almazan
Sr. Partner Solutions Engineer
salmazan@cloudera.com
Tim Spann
DIM Developer Advocate
tspann@cloudera.com

3
© 2023 Cloudera, Inc. All rights reserved.
TODAY’S LEAD
Who am I?
@PaasDev
DZone Zone Leader and Big Data MVB
Princeton and NYC Future of Data Meetups
ex-Pivotal Field Engineer ex-StreamNative ex-PwC
https://github.com/tspannhw https://twitter.com/PaaSDev
https://www.datainmotion.dev/
https://medium.com/@tspann
Principal Data-in-Motion Developer Advocate

© 2023 Cloudera, Inc. All rights reserved. 4
Future of Data - NYC / Princeton + Virtual
@PaasDev
https://www.meetup.com/futureofdata-princeton/
https://www.meetup.com/futureofdata-newyork/
From Big Data to AI to Streaming to LLM to Cloud to
Analytics to NLP to Fast Data to Machine Learning to
Microservices to ...

5
Streaming Change Data Capture (CDC) 3+ Unique Ways
In this next session,
learn how to use Debezium with Flink, Kafka, and NiFi for Change Data Capture using two different mechanisms: Kafka
Connect and Flink SQL.
With the virtual nature of today's world, streaming data is more critical than ever. Join Cloudera Chief Data-In-Motion Principal,
Tim Spann, and Partner Solution Engineer, Salvador Alamazan as they look closely at key CDC use cases, discuss why
Debezium is the best option for handling CDC and use examples to show you how to demonstrate value.
This is a must-attend experience!
https://medium.com/cloudera-inc/cdc-not-cat-data-capture-e43713879c03
https://dzone.com/articles/streaming-change-data-capture-data-two-ways

7
What is Change Data Capture
Full Fidelity vs Point in Time

8
Why Change Data Capture?
Most Common Use Cases - Distribution or Synchronization
Analytics
{Distribution}
Full Fidelity Analytics
Oﬄine, Look at any point in
time
Operational Data Store
{Distribution}
Prevent load on
operational databases by
replicating the database
for reporting queries to run
in isolation
Migrations
{Synchronization}
Keeping two databases in
synchronization, existing
and new systems need to
coexist for some time

CDC ENGINE SELECTION
HOW TO DO IT?

Already using Kafka? Already using NiFi? Need for Fast Flink?
Simple setup for many tables
Want metadata augmented data
Don’t need low latency?
Visual monitoring
Easy manual scaling
Easy to combine with NiFi
Debezium
Simple JDBC queries?
Transform individual records?
Want easy development with UI?
Lots of small ﬁles, events, records, rows?
Continuous stream of rows
Support many different sources
Debezium coming
Strong control of table and joins
Want high Throughput?
Want Low Latency?
Want Advanced Windowing and State?
Automatic records immediately
Pure SQL
Debezium
Kafka Connect, NiFi, Flink? Which engine to choose? Or All 3?

CDC ARCHITECTURE - Using FLaNK to pull the data out of anything in near-real time
INGEST PREPARE PUBLISH
DATA SOURCES
Internal Users
(After Sales)
External
Systems
ENTERPRISE
LAKEHOUSE
CAPABILITY VIEW
INGESTION
ENTERPRISE DATA
MESSAGE HUB
STORAGE
BATCH
MANAGEMENT
STREAM
CONSUMPTION
Closed Loop
Systems
SQL Stream Builder
Machine Learning
Data Visualization
Workload Manager
watsonx.data

12
Data Distribution as a Universal, Hybrid, Multi-Cloud Data Service
Universal Data Distribution Service
(Ingest, Transform, Deliver)
Ingest
Processors
Ingest Gateway
Router, Filter &
Transform Processors
Destination
Processors
Cloud Business Process Services*
Log Data Sources
Laptops /
Servers Security
Agents
IOT Devices App Logs
Mobile Apps
Cloud Data Analytics/ Service *
On-Prem Data Sources Cloud Warehouse
(Cloudera DW)
Big Data Cloud Services
Multi-Cloud Data Distribution Service that Solves the First & Last Mile Problem for the Modern Data Stack

CDC with SQL Stream Builder
(Flink SQL)

14
Streaming CDC with Cloudera SQL Stream Builder (Flink SQL)
https://github.com/tspannhw/FLaNK-CDC/blob/main/flinkcdc.MD

https://docs.cloudera.com/csa/1.10.0/how-to-ssb/topics/csa-ssb-cdc-connectors.html
CDC with Debezium and Flink
SQL Stream Builder with Flink SQL

CDC with Debezium and Flink
SQL Stream Builder with Flink SQL

CREATE TABLE `postgres_cdc_newjerseybus` (
`title` STRING,
`description` STRING,
`link` STRING,
`guid` STRING,
`advisoryAlert` STRING,
`pubDate` STRING,
`ts` STRING,
`companyname` STRING,
`uuid` STRING,
`servicename` STRING
) WITH (
'connector' = 'postgres-cdc',
'database-name' = 'tspann',
'hostname' = '192.168.1.153',
'password' = 'tspann',
'decoding.plugin.name' = 'pgoutput',
'schema-name' = 'public',
'table-name' = 'newjerseybus',
'username' = 'tspann',
'port' = '5432'
);
Flink SQL Tables - Debezium CDC From Database Tables

Flink SQL Tables - Upsert to Kafka Topics
CREATE TABLE ùpsert_kafka_newjerseybus` (
`title` String,
`description` String,
`link` String,
`guid` String,
àdvisoryAlert` String,
`pubDate` String,
`ts` String,
`companyname` String,
ùuid` String,
`servicename` String,
èventTimestamp` TIMESTAMP(3),
WATERMARK FOR èventTimestamp` AS èventTimestamp` - INTERVAL '5' SECOND,
PRIMARY KEY (uuid) NOT ENFORCED
) WITH (
'connector' = 'upsert-kafka',
'topic' = 'kafka_newjerseybus',
'properties.bootstrap.servers' = 'kafka:9092',
'key.format' = 'json',
'value.format' = 'json'
);

21
https://github.com/tspannhw/FLaNK-CDC/blob/main/kafkacdc.md
Streaming CDC with Cloudera Streams Messaging Manager (Kafka)

22
CDC with Debezium and Kafka
Kafka Connect

23
What is it?
• Connect data sources (or sinks) with Kafka
– Ideal for moving large amounts of data into or out of Kafka
– In a reliable manner
– In a performant way
• A tool built upon Apache Kafka
• Present since end of 2015
• Makes easier to implement the most common use case, earlier (presumably)
implemented with Producer/Consumer API
Kafka Connect

24
Basic Concepts
Kafka Connect
● Connect framework uses “Workers” (Java processes) to do data processing.
● Workers Run Connectors
● Workers can be organized into a Connect cluster.

25
KConnect
Connectors
• New Connectors
• CDC Debezium Connectors
• SMM UI Integration
• Reuse your Kafka
Infrastructure
• Enterprise Security for
Secrets, Authentication
and Authorization
Sources
● ActiveMQ (via JMS)
● MQTT
● Syslog over TCP
● JDBC
● JMS
● HTTP
Sinks
● AWS S3
● ADLS
● Kudu
● HDFS
● HTTP
CDC Connectors
● MySQL
● PostgreSQL
● Oracle
● MS SQL Server
● IBM DB2

29
Apache NiFi + Apache Kafka
Process Debezium Kafka Formats From Kafka Connect and others

30
Apache NiFi to Databases
SQL and MySQL CDC

31
Apache NiFi to Databases
SQL Query Access

33

35
Cloudera Streams
Processing -
Community Edition
• Kafka, KConnect, SMM, SR,
Flink, and SSB in Docker
• Runs in Docker
• Try new features quickly
• Develop applications locally
● Docker compose ﬁle of CSP to run from command line w/o any
dependencies, including Flink, SQL Stream Builder, Kafka, Kafka
Connect, Streams Messaging Manager and Schema Registry
○ $> docker compose up
● Licensed under the Cloudera Community License
● Unsupported
● Community Group Hub for CSP
● Find it on docs.cloudera.com under Applications

Open Source Edition
• Apache NiFi in Docker
• Runs in Docker
• Try new features
quickly
• Develop applications
locally
● Docker NiFi
○ docker run --name nifi -p 8443:8443 -d -e
SINGLE_USER_CREDENTIALS_USERNAME=admin -e
SINGLE_USER_CREDENTIALS_PASSWORD=ctsBtRBKHRAx69EqUgh
vvgEvjnaLjFEB apache/nifi:latest
● Licensed under the ASF License
● Unsupported
https://hub.docker.com/r/apache/niﬁ

Cloudera Edge2AI Workshop - CDC - Debezium
https://github.com/asdaraujo/edge2ai-workshop/blob/trunk/workshop_cdc.adoc

https://medium.com/@tspann/cdc-not-cat-data-capture-e43713879c03

40
References
https://medium.com/@tspann/ingesting-events-into-dockerized-ibm-db2-jdbc-with-apache-ni
ﬁ-f0ca452d1351
https://community.cloudera.com/t5/Community-Articles/RDBMS-to-Hive-using-NiFi-small-m
edium-tables/ta-p/244677
https://community.cloudera.com/t5/Community-Articles/MySQL-CDC-with-Kafka-Connect-D
ebezium-in-CDP-Public-Cloud/ta-p/345321
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/kafka-connect/topics/kafka-connec
t-connector-debezium-db2.html
https://docs.cloudera.com/csa/1.10.0/how-to-ssb/topics/csa-ssb-cdc-connectors.html
https://medium.com/cloudera-inc/building-a-stateful-streaming-intrusion-detection-system-
with-sql-stream-builder-4667c87f347f

41
Streaming Resources
• https://dzone.com/articles/real-time-stream-processing-with-hazelcast-an
d-streamnative
• https://ﬂipstackweekly.com/
• https://www.datainmotion.dev/
• https://www.ﬂankstack.dev/
• https://github.com/tspannhw
• https://medium.com/@tspann
• https://medium.com/@tspann/predictions-for-streaming-in-2023-ad4d739
5d714
• https://www.apachecon.com/acna2022/slides/04_Spann_Tim_Citizen_Str
eaming_Engineer.pdf

42
CDC
https://community.cloudera.com/t5/Community-Articles/Incrementally-Streaming-RDBMS-Data-to-Your
-Hadoop-DataLake/ta-p/247927
https://community.cloudera.com/t5/Community-Articles/QADCDC-Our-how-to-ingest-some-database-ta
bles-to-Hadoop-Very/ta-p/245229
https://community.cloudera.com/t5/Community-Articles/Ingesting-RDBMS-Data-As-New-Tables-Arrive-
Automagically-into/ta-p/246214
https://community.cloudera.com/t5/Community-Articles/Ingesting-Golden-Gate-Records-From-Apache-
Kafka-and/ta-p/247557
https://community.cloudera.com/t5/Community-Articles/Simple-Change-Data-Capture-CDC-with-SQL-
Selects-via-Apache/ta-p/308390

43
CDC
https://community.cloudera.com/t5/Community-Articles/RDBMS-to-Hive-using-NiFi-small-medium-table
s/ta-p/244677
https://community.cloudera.com/t5/Community-Articles/Simple-Change-Data-Capture-CDC-with-SQL-
Selects-via-Apache/ta-p/308376

https://community.cloudera.com/t5/Community-Articles/Incrementally-Streaming-RDBMS-Data-to-Yo
ur-Hadoop-DataLake/ta-p/247927
https://community.cloudera.com/t5/Community-Articles/Ingesting-RDBMS-Data-As-New-Tables-Arriv
e-Automagically-into/ta-p/246214
https://community.cloudera.com/t5/Community-Articles/Incremental-Fetch-in-NiFi-with-QueryDatabas
eTable/ta-p/247073
https://community.cloudera.com/t5/Community-Articles/Simple-Change-Data-Capture-CDC-with-SQL
-Selects-via-Apache/ta-p/308390
Cloudera Data Flow / Apache NiFi

https://community.cloudera.com/t5/Community-Articles/Change-Data-Capture-CDC-with-Apache-NiF
i-Part-1-of-3/ta-p/246623
Cloudera Data Flow / Apache NiFi

https://community.cloudera.com/t5/Community-Articles/MySQL-CDC-with-Kafka-Connect-Debezium-
in-CDP-Public-Cloud/ta-p/345321
CDC Debezium KConnectors for PostgreSQL, MySQL, SQL Server, DB2, and Oracle
Cloudera Streams Messaging / Apache Kafka Connect

PartnerSkillUp_Enable a Streaming CDC Solution

Recommended

More Related Content

Similar to PartnerSkillUp_Enable a Streaming CDC Solution (20)

More from Timothy Spann (20)

Recently uploaded (20)

PartnerSkillUp_Enable a Streaming CDC Solution