Cloud-Native Streaming Platform: Running Apache Kafka on PKS (Pivotal Container Service)

Cloud-Native
Streaming Platform:
Apache Kafka
Meets Kubernetes
@gamussa #BOSDataDay @confluentinc

@@gamussa #BOSDataDay @confluentinc
#devkafkaops

https://twitter.com/kelseyhightower/status/963413508300812295

https://twitter.com/kelseyhightower/status/963414038603427840

Don’t despair…
“… not even over the fact that you don't
despair. Just when everything seems over
with, new forces come marching up, and
precisely that means that you are alive”
Franz Kafka

Kafka Streaming Architecture
Fundamentals

Shard data to get scalability
Producer (1) Producer (2) Producer (3)
Cluster of
machines
Partitions live on
different machines
Messages are sent to
different partitions

Linearly Scalable Architecture
Single topic:
- Many producers machines
- Many consumer machines
- Many Broker machines
No Bottleneck!!
Producers
Consumers

Replicate to get fault tolerance
replicate
msg
msg
leader
Machine A
Machine B

Replication provides resiliency
A ‘replica’ takes over on machine failure

High-level Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

Bare minimum
Zookeeper NodesKafka Brokers

Workloads Deployment

Helm Charts

https://cnfl.io/helm_video

Kafka deployment checklist
PVC for Storage
StatefulSet for 3-node zk
Optional Pod Anti-Affinity to spread the
ZK ensemble across nodes
Headless Service
ConfigMap for Prometheus JMX exporter
Uses ZK Headless Svc
PVC for Storage
StatefulSet for n-node Kafka
A group of NodePort Services for
external traffic
ConfigMap for Prometheus JMX exporter

Meet Kubernetes Operator

Kubernetes Operator
Embedded with operational knowledge
of both data software and
Kubernetes
Backup/restore
Scale up/down
Rebalance data
Regular health checks

Controller
Brain behind Kubernetes
resources
e.g. replication
controller, namespace
controller etc.

Custom Resource Definition(CRD)
Extend existing
Kubernetes API StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
StatefulSet ReplicaSet
Custom
Resource
Instance
...
...

Usually works together
Custom Controller StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
Custom
Resource
Instance
...
...

Users can create and
access Customer
Resources with
kubectl, just as they
do for built-in
resources like pods.
StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
Custom
Resource
Instance
...
...

Operator
Deploy and Manage your production
streaming platform with Confluent
Operator.
Automated Provisioning
Platform Operations
Resiliency
Monitoring

Confluent Platform Reference Architecture
Each Confluent Platform
component has specific
characteristics:
Security (SSL certificates)
DNS names and zones
Host selection
Fault tolerance
Scaling
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

Confluent Operator: Automated Provisioning
Kafka Pod Kafka Pod Kafka Pod
Storage
Load Balancer

Confluent Operator: Scale Horizontally
Automate scaling:
Spin up new broker pod(s)
Distribute partitions to the new
broker(s)
Determine balancing plan
Execute balancing plan
Monitor resources

Confluent Operator: Rolling Upgrade
Automated rolling upgrade with no
downtime for Kafka.
Stop broker
Wait for leader election to
complete
Start broker with new version
Wait for zero under-replicated-
partitions
Repeat

Will it fly? Let’s see

Confluent Operator
Automate provisioning
Monitor SLAs through
Confluent Control Center
or Prometheus
Scale your Kafkas and
CP clusters elastically
Operate at scale with
enterprise support
from Confluent

Advanced use cases
vs.

Don’t despair!

Resources and Next Steps
https://cnfl.io/cp-helm
https://cnfl.io/k8s
https://slackpass.io/confluentcommunity
#kubernetes
https://cnfl.io/helm_video

Thanks!
@gamussa
viktor@confluent.io

Cloud-Native Streaming Platform: Running Apache Kafka on PKS (Pivotal Container Service)

More Related Content

What's hot (6)

Similar to Cloud-Native Streaming Platform: Running Apache Kafka on PKS (Pivotal Container Service) (20)

More from Carlos Andrés García (8)

Recently uploaded (20)

Cloud-Native Streaming Platform: Running Apache Kafka on PKS (Pivotal Container Service)