Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd

1Apache Kafka and Service Mesh (Envoy / Istio) – Kai Waehner
Event Streaming Platform and Service Mesh
Cloud-Native Apache Kafka with Kubernetes, Envoy and Istio
Kai Waehner
Technology Evangelist
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de

Key Takeaways
• Apache Kafka decouples services, including event streams and request-response
• Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
• Service Mesh helps with security and observability at ecosystem / organization scale
• Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
+

Agenda
• Motivation, Challenges, Requirements of Microservices
• Apache Kafka - The Event Streaming Platform for Microservices
• Kubernetes for Cloud-Native Microservices
• Service Mesh
• Service Proxy (aka Data Plane)
• Control Plane
• Kafka and Service Mesh
• Service Mesh Implementation with Kafka, Kubernetes, Envoy, Istio

Agenda
• Service Mesh
• Control Plane

Business Digitalization Trends are Driving the Need to Process
Events at a whole new Scale, Speed and Efficiency
Mobile Cloud Microservices Internet of Things Machine Learning
The world has changed!

Microservices to the rescue?
• Significant Operations Overhead
• Substantial DevOps Skills Required
• Implicit Interfaces
• Duplication Of Effort
http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
• Distributed System Complexity
• Asynchronicity Is Difficult
• Testability Challenges

Key Requirements for Microservices
Decoupled
Flexible
Operationally Transparent
Data Aware
Elastic

Agenda
• Service Mesh
• Control Plane

The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Apache Kafka—The Rise of an Event Streaming Platform

Apache Kafka at Scale at Tech Giants
> 4.5 trillion messages / day > 6 Petabytes / day
“You name it”
* Kafka Is not just used by tech giants
** Kafka is not just used for big data

Confluent - Business Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)

Apache Kafka - A Distributed Commit Log
Writers
Kafka
cluster
Readers

Kafka Topics
my-topic
my-topic-partition-0
broker-1
broker-2
broker-3

P
Producing to Kafka
Time
C2 C3C1

Partition Leadership and Replication
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4

Schema are about how teams work together

A quick change of the timestamp format…

… breaks things!

APIs between services are Contracts
In Stream Processing World – Event Schemas ARE the API
Governance in a Streaming Architecture

Confluent Schema Registry

Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design (DDD) for your Event Steaming Platform
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!

Decoupled
Flexible
Data Aware
Elastic

Agenda
• Service Mesh
• Control Plane

Cloud-Native Platforms in last five years

Kubernetes won the battle!

Cloud-Native Deployment leveraging Kubernetes

Evolution of Kafka DevOps
Shell scripts
Ansible/Chef Docker
Kubernetes

Kafkaesque world of Kafka on Kubernetes

29
Kafka on Kubernetes –
It’s tricky L
• Translating an existing architecture to Kubernetes
• Failover handling and data balancing
• Communication between ZooKeeper, Kafka Brokers, Clients (Java,
REST, Connect, KSQL), Schema Registry, etc.
• External access from / to outside Kubernetes cluster
• Persistent storage options on prem and in the cloud
• Security configuration
• Rolling upgrades
• Etc.

Kafka Operator for Kubernetes
The Operator pattern for Kubernetes aims to capture the key aim of a human operator
who is managing a service or set of services. Human operators who look after specific
applications and services have deep knowledge of how the system ought to behave,
how to deploy it, and how to react if there are problems.
People who run workloads on Kubernetes often like to use automation to take care of
repeatable tasks. The Operator pattern captures how you can write code to automate a
task beyond what Kubernetes itself provides.
Some Kafka Operators:

Confluent’s Kubernetes Journey building “Confluent Cloud”
05/2017
Confluent Cloud
Early Access
2016
Confluent Cloud
Development
11/2017
Confluent Cloud
GA (AWS)
07/2019
Confluent Operator GA
(Confluent Platform)
2019
Confluent Cloud GA
on AWS, GCP, Azure

Confluent Operator
Deployment and management automation for
Confluent Platform on Kubernetes
Including Apache Kafka, Zookeeper, Schema Registry,
Connect, Control Center, Replicator, KSQL
For organizations standardized on Kubernetes as
platform runtime
Operationalizes years of experience running Kafka on
Kubernetes on premises or the leading public clouds
Confluent Platform
Confluent Operator
Kubernetes
AWS Azure GCP
RH OpenShift Mesosphere Pivotal
On-Premises Cloud
Docker Images
Automate Deployment of Confluent Platform on Kubernetes on Any Platform at Any Scale

Confluent Operator enables you to:
Automate provisioning of
Kafka pods and security
configuration in minutes
Monitor SLAs through Confluent
Control Center or Prometheus
Scale Kafka elastically &
Automate rolling updates
Built on our first hand knowledge
of running Confluent at scale
Cloud-Native Deployment of Kafka and Confluent Platform

Kubernetes
Cluster K8 NodeK8 NodeK8 Node
Replicator
Pod
C3 Pod SR Pod
K8 NodeOperator
Kafka
Pod
ZK Pod
Persistent Volumes
(AWS EBS, GCE Persistent Disk, Local Persistent Volume, etc.)
External
Access
Load
Balancers
Configurations
ConfigMapsKSQL Pod
REST Proxy
Pod
Confluent Operator Deployment

Decoupled
Flexible
Data Aware
Elastic

Agenda
• Service Mesh
• Control Plane

Excursus: Kubernetes Pod
“pod == small herd of aquatic mammals, esp. of whales or dolphins”
https://geekdudes.wordpress.com/2019/07/14/kubernetes-creating-pods-on-windows-10/

Sidecar Pattern
38
Components of the application, deployed in a separate container to provide
isolation and encapsulation. This pattern allows applications to be composed of
heterogeneous components.

Service Mesh
A microservice pattern to move visibility, reliability, and
security primitives for service-to-service communication
into the infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
Data Plane
Touches every packet/request in the system.
Responsible for service discovery, health
checking, routing, load balancing,
authentication/authorization, and observability.
Control Plane
Provides policy and configuration for all of the
running data planes in the mesh. Does not touch
any packets/requests in the system. The control
plane turns all of the data planes into a distributed
system.

Service Mesh
Out of process architecture
• Self contained process
• Run alongside every application server
• Application sends and receives messages to
and from localhost and is unaware of the
network topology
Benefits
Compared to “fat client proxy libraries” like Finagle
(Twitter), Hystrix (Netflix), Stubby (Google):
• Works with any application language (Java,
C++, Go, PHP, Python, etc.)
• Can be deployed and upgraded quickly across
an entire infrastructure transparently
https://www.infoq.com/articles/linkerd-v2-production-adoption/

Agenda
• Service Mesh
• Control Plane

Excursus: Load Balancing and Proxy at L3/L4 vs. L7 of OSI Model
https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy
https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236
L3/L4 vs. L7 is
not always
the right
abstraction level!

Excursus: Load Balancing and Proxy at L3/L4 vs. L7
https://www.envoyproxy.io/docs/envoy/latest/intro/what_is_envoy
https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236
Example: Envoy Proxy Features
L3/L4 filter architecture
HTTP L7 filter architecture
HTTP L7 routing
gRPC support
MongoDB L7 support
DynamoDB L7 support
Kafka L7 support (Pull request merged in May 2019)
Service discovery and dynamic configuration
Health checking
Advanced load balancing
Front / edge proxy support
Observability (stats, tracing)

Proxy
Proxy
Proxy
Proxy
Proxy
Service Proxy as Sidecar

Proxy
I have a new IP
now.
Who
cares?
I magically
know all about
it!
Example – Service Proxy as Inbound Sidecar

Proxy
I can recover from
errors without
drowning
Error?
No worries!
Lets retry every
millisecond forever
LOL. I’m
dropping 99%
of the retries.
Example - Service Proxy as Outbound Sidecar

Service Proxy Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service Mesh
provide in a distributed Microservice architecture!” Matt Klein

Why Lyft built “envoy” Proxy
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
Developers should be able
to spend their time on
writing business applications

Lyft today with “envoy” Proxy
100% (!!!) communication coverage - Everything talks through Envoy Proxies
à Make monitoring, debugging, firefighting as consistent as possible
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
Service Mesh to the rescue:
• Abstract network from
application developers
• Get operational transparency
and more flexibility

Example: Advanced Load Balancing with Linkerd
https://linkerd.io/2016/03/16/beyond-round-robin-load-balancing-for-latency/
Since latency and failure are often
tied together in distributed systems
via timeouts, we can also express the
results in terms of failure.
If the caller of our system used a
timeout of 1 second, its success rate
would be approximately 95% with
round robin, 99% with least loaded,
and 99.9% with peak EWMA
(exponentially-weighted moving
average) - a significant difference.

Agenda
• Service Mesh
• Control Plane

Control Plane + Proxy as Sidecar = Service Mesh
(Human Control Plane)
https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc

Control Plane + Proxy as Sidecar = Service Mesh
(Advanced Service Mesh Control Plane)
https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc
“Ultimately, the goal of a control plane
is to set policy that will eventually be
enacted by the data plane.”

Example for Control Plane - Istio Architecture
Pilot: Service discovery and
configuration of Envoy sidecar proxies
Mixer (Istio-Policy and Istio-Telemetry):
Enforcement of usage policies and
gathering of telemetry data
Ingress / Egress Gateway: Points for
traffic to ingress or exit from outside
the cluster
Citadel: Automation of key and
certificate management
Galley: Configuration management
services

Support for the Three Pillars of Observability

Decoupled
Flexible
Data Aware
Elastic

Service Mesh Interface (SMI)
https://www.infoq.com/presentations/service-mesh-interface

Agenda
• Service Mesh
• Control Plane

Service Mesh and Event Streaming Platform
59
Request-Response Events Streams
- Low latency
- Typically sync
- Point-to-point
- “Bespoke API”
- e.g. HTTP, gRPC
- Continuous processing
- Often async
- Event driven
- General-purpose events
- e.g. Apache Kafka
Traditionally, these are two different paradigms:
Please…
No REST vs.
Streaming FUD!
Most architectures need request-response and event streams!

Why not use Service Mesh and Event Streaming Platform together?
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy

Kafka Connect
Kafka Cluster
CRM Integration
Clients and Servers are Independent (including their Ops Teams)
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy Proxy Proxy
Proxy Proxy Proxy
Control
Plane

Proxy
I am somehow
getting events
from Kafka
I’m using REST to
talk to a service
I’m really re-
directing
events to Kafka
Introduce
Vision #1: Using Service Mesh to Hide Kafka
Gwen Shapira (June 2018): Visionary ideas about Kafka and Service Mesh
https://www.youtube.com/watch?v=Fi292CqOm8A

Proxy
I am using REST too!
Kafka? Never heard of
her.I’m using REST to
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
Vision #2: Kafka as Part of Control Plane

Vision #3: Kafka as a Service in a Mesh
Proxy
Proxy
Proxy
Proxy
Proxy
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)
Kafka
Protocol
(TCP)

Vision #4: Front Kafka (-as-a-Service)
P
R
O
X
Y
P
R
O
X
Y

(Potential) Features for Kafka + Service Mesh Implementation
Protocol conversion from HTTP /
gRPC to Kafka
• Tap feature to dump to a Kafka stream
• Protocol parsing for observability (stats,
logging, and trace linking with HTTP
RPCs)
• Shadow requests to a Kafka stream
instead of HTTP / gRPC shadow
• Integrate with Kafka Connect and its
whole ecosystem of connectors
Validation of Events
• Serialization format (JSON,
Avro, Protobuf, etc.)
• Message schema
• Headers, attributes, etc.
Security
• SSL Termination
• Mutual TLS (mTLS)
• Authorization
Proxy features
• Dynamic Routing
• Rate limiting at both the L4 connection
and L7 message level
• Filter, add compression, …
• Automatic topic name conversion (e.g.
for canary release or blue/green
deployment)
Monitoring and Tracing
• Request logs and stats
• Data lineage / audit log
• Audit log by taking request logs and
enriching them with the user info.
• Client specific metrics (Byte rate per
client id / per consumer groups,
versions of the client libraries,
consumer lag monitoring for the
entire data center)

Put logic outside Kafka vs. make deployment as simple as possible
Server-side Schema Validation on Kafka Broker
Goal: Tiered Storage and Autoscaling

Agenda
• Service Mesh
• Control Plane

Service Mesh Implementation
Various options for a Service Mesh implementation; examples à
Some examples with Kafka, Kubernetes*, Envoy**, Istio:
• L4: Filter on Kafka Client side (rate limiting, mTLS, etc.)
• L4: Filter on Kafka Broker side (rate limiting, mTLS, etc.)
• L7: Confluent REST Proxy on Server side
• L7: Envoy’s Kafka Protocol Filter
• L7 Filter + Routing
• L7 Observability
• Many more Kafka-specific features possible
• L7: Custom proxy implementation
• Example: https://github.com/travisjeffery/kafka-proxy
* Kubernetes is assumed as de facto standard
** Envoy has best Kafka integration (in September 2019)

L4 Example: Kafka + Istio @ Banzai Cloud
https://banzaicloud.com/blog/kafka-on-istio-performance/

L7 Example: Kafka + Confluent REST Proxy
Envoy
Proxy
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
Confluent
REST Proxy
I support only TCP!
HTTP
HTTP

Kafka Support in Envoy (Pull Request Merged in May 2019)
https://github.com/envoyproxy/envoy/issues/2852
https://github.com/envoyproxy/envoy/pull/4950

Kafka Support in Istio? (August 2019)
• Before PR #4950, Envoy treats Kafka as TCP, so that Istio-TCP-rules will apply (already).
• With PR #4950, Envoy can do some more fancy things and get stuff like number-of-messages in
telemetry from Kafka semantics.
• Now the 2nd part here is Istio, which needs a new vocabulary to be able to configure Envoy.
Think of ‘VirtualService’ and ‘DestinationRule’ for Kafka (or messaging in a more global sense).
• TLDR: L4 works in Istio; L7 needs some new PRs in Istio project, too!
https://istio.io/docs/tasks/traffic-management/tcp-traffic-shifting/

L7 Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
talk to a service
I’m proxying
REST.
And also logging
stuff to Kafka
HTTP
TCP
(Kafka Protocol)

Decoupled
Flexible
Data Aware
Elastic

Event Streaming Platform and Service Mesh
A Match Made In Heaven
+ =

Key Takeaways
• Apache Kafka decouples services, including event streams and request-response
• Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
• Service Mesh helps with security and observability at ecosystem / organization scale
• Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
+

Kai Waehner
Technology Evangelist
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd

Recommended

More Related Content

What's hot (20)

Similar to Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd (20)

More from Kai Wähner (20)

Recently uploaded (20)

Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd