Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | Fabian Martinez and Hugo Guerrero, Red Hat

Event-driven APIs & schema governance for Apache
Kafka
Apicurio Registry
Fabian Martinez
Senior Software Quality Engineer
@famargon13
Hugo Guerrero
APIs & Messaging Developer Advocate
@hguerreroo
1

EVENT-DRIVEN APIS & SCHEMA GOVERNANCE
Who are we?
2
Fabian Martinez (@famargon13)
● Senior Software Quality Engineer @ Red Hat
● Java, Microservices & Kubernetes
● Good coffee lover
Hugo Guerrero (@hguerreroo)
● Developer Advocate @ Red Hat
● APIs & Messaging Specialist
● Food, Travel & History Enthusiast
● Writes at https://hellokube.dev

What is the situation?
3
Distributed Decoupled Connected

4
Microservices Request / Response Events

5
Microservices Request / Response Events

The Traditional Workflow
6
Project Planning
The business experts spend time writing up a
project plan for a series of services and
potentially a user interface
Send to Developers
The various development teams attempt to
interpret the project plan and implement their
code so that they are compatible
Iterate and Integrate
The developer teams iteratively try to resolve
integration issues and spend loads of time
ensuring compatibility

Kafkaesque
7
Photo credit: dbeck03 on VisualHunt
Endpoint Documentation ?
Data Format and Validation ?
Infrastructure Access?

8
Why Contract-First?
● Because you want to allow people to work independently
● Because you want to ensure consistency
● Because you need strong guarantees about service contracts
● Because you, your team, your customers, and your partners can collaborate
● Because you can save time by using code generators and testing tooling

How does it work on REST APIs?
9
Source:
https://swagger.io/docs/specification/about/
https://www.apicur.io/
https://openapi-generator.tech/
https://microcks.io/
Build an API Specification
Using a tool like OpenAPI,
write an API specification FIRST
Publish the API Specification
Using a tool like Swagger or Apicur.io,
publish the API specification where others
have access and can collaborate
Generate Code, Mocks and Testing
Using a tool like OpenAPI Generator to create
the API stubs for both client and server
applications. You can also generate “mock”
services with Microcks.

Event-driven Secret Weapon
10
An open source initiative that seeks to improve the
current state of Event-Driven Architectures.
Created to describe Event-driven APIs using MQTT,
AMQP, Apache Kafka, ...
“Sister” specification to OpenAPI: same base syntax,
use JSON Schema underhood.
Allows definition of examples
AsyncAPI
https://www.asyncapi.com/

Unified API contracts
11
POST /register
{
"fullName":"John Doe",
"email":"john@example.io",
"age":36
}
TOPIC user-registration
{
"id":"38102178-dbe5-4e81-aa65-8f6df1be4d44",
"fullName":"John Doe",
"email":"john@example.io",
"age":36,
"sendAt":"1603786974956",
}
OpenAPI 3
Contract
AsyncAPI 2
Contract

AsyncAPI Anatomy
12
https://www.asyncapi.com/docs/getting-started/coming-from-openapi
Info: metadata and documentation for this asynchronous API: version, title,
description, contact and licensing terms
Servers: list of brokers with binding details (URLs and security protocols)
Channel Item: destination that will be used by the API to publish or consume
events. Destination may be mapped to a topic, queue, address name
depending on the protocol binding.
Operation: whether the application using the AsyncAPI should subscribe or
produce messages. Holds its own descriptive metadata
Message: the content-type and structure of message headers and payload.
May reference schemas of different types (AsyncAPI, JSON, Avro). May
contain examples!
Components: Common definitions (schemas, traits, messages) like in
OpenAPI v3

AsyncAPI Anatomy
13
https://www.asyncapi.com/docs/getting-started/coming-from-openapi
Info: metadata and documentation for this asynchronous API: version, title,
description, contact and licensing terms
Servers: list of brokers with binding details (URLs and security protocols)
Channel Item: destination that will be used by the API to publish or consume
events. Destination may be mapped to a topic, queue, address name
depending on the protocol binding.
Operation: whether the application using the AsyncAPI should subscribe or
produce messages. Holds its own descriptive metadata
Message: the content-type and structure of message headers and payload.
May reference schemas of different types (AsyncAPI, JSON, Avro). May
include examples!
Components: Common definitions (schemas, traits, messages) like in
OpenAPI v3

Schemas are events
contracts
EVENT-DRIVEN
APIS
&
SCHEMA
GOVERNANCE
14

Contract-First API Development
15
Why is it important to use API contracts in Apache Kafka?
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer

17
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
Consumer
Consumer
v2
v2
v2
v1

18
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
Consumer
Consumer
v2
v2
v2
v1

19
Producer
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Send
Data
Retrieve
Data
Consumer
Consumer
Consumer
v2
v2
v2
v1

What issues needs to be addressed in a
Kafka world?
● Kafka is not aware of data types or
schema formats.
● API schemas are subject to change.
● Central registry where the data schemas
are stored and accessible.
20
Photo credit: Harald Groven on VisualHunt

Registry for Schemas to
the rescue
EVENT-DRIVEN
APIS
&
SCHEMA
GOVERNANCE
21

Using the Registry for Schema Management
22
Producer Consumer
Deserializer
Serializer
Registry
| | | | | | |
| | | | | | | |
| | | | | |
Apache Kafka
Get or
Register
Schema
by Id
Retrieve
Schema
by Id
Topic B
(JSON)
Topic C
(Protobuf)
Topic A
(Avro)
Send
Serialized
Data
Retrieve
Serialized
Data

Optional section marker or title
What are the required
capabilities of the registry?
23
The registry needs to address
the following 3 pillars for a
successful management
You need to be able to manage all the different types of
artifacts, be able to browse, download and document in
single place.
Artifact Management
Support for different types of data structures, like Apache
Avro, Google Protocol Buffers, or JSON Schemas.
Data Formats
Artifacts follow content evolution and are controlled by
enabling content rules, including Validity and Compatibility.
Versioning

Apicurio Registry
@apicurio
https://www.apicur.io/registry/
https://github.com/Apicurio/apicurio-registry
24

Apicurio Registry
25
Use Cases
Source:
API specification registry for
API consumers
Shared Data Types
API Designs
Shared data types (schemas) across
API and event driven architectures
Schema Registry
Schema registry for Kafka
serializers/deserializers

● Native support for multiple Schema formats:
○ Apache Avro
○ JSON Schema
○ Protobuf (protocol buffers)
○ OpenAPI
○ AsyncAPI
○ ...
● Version control
○ Schemas versioning management
● Compatibility and Validity enforcement rules
○ Backwards compatibility enforcement, among others
○ Schema validity enforcement, among others
Apicurio Registry
26
Features
Source:

● Paired with lots of tools
○ Web UI
○ Kafka Serializer / Deserializer (Serdes)
○ Kafka Connect Converters
○ Maven Plugin for schema management
○ Kubernetes Operator
○ CLI tool
Apicurio Registry
27
Features
Source:

An Improved Registry:
Service Registry 2.0 for
Red Hat Integration
EVENT-DRIVEN
APIS
&
SCHEMA
GOVERNANCE
28

● Support for Artifact groups
○ New v2 REST API with top level groupings support
● Improved Serdes libraries
○ Independent libraries for each data format
○ Avro, Protobuf and Json Schema
○ Simplified configuration
○ Lots of powerful configuration options
Service Registry 2.0
29
New Features
Source:

● Authentication support
○ Integration with the OIDC protocol for securing the Registry
API
● Support for event-driven integrations
○ Allowing for custom or third party integrations
○ Webhooks
○ Kafka
30
New Features
Source:

● Implementation preview of the CNCF Schema
Registry API.
● Part of the CloudEvents Working Group efforts
for standardization and interoperability.
31
CNCF Schema Registry API support
Source:
https://github.com/cloudevents/spec/blob/master/schemaregistry/schemaregistry.md

32
Source Code: https://github.com/maschind/apicurio-kafka-demo
Photo credit: nickboos on Visualhunt

Red Hat Integration
33
Source
▸ Change Data Capture
with Debezium
API Management
Events & Messaging
Enterprise Integration
Data Integration
▸ Comprehensive connectors
▸ Microservices orchestration
▸ Data Transformation
▸ Low-code iPaaS
▸ Serverless Composition with
Camel K
▸ JMS Message Broker
▸ Wide Area Routing
▸ Data Streaming with
Apache Kafka
▸ Self-service messaging
▸ API Manager
▸ API Gateway
▸ Istio Service Mesh
Adapter
▸ Service Registry
▸ API Designer
Tooling & Metadata

34
Kafka Cluster
Mirror Maker 1
Entity Operator
(Topic, User)
Cluster Operator
Zookeeper
Cluster
HTTP Bridge
Kafka Connect
Cluster
Debezium
Connectors
Camel Kafka
Connectors Other Connectors
Service Registry
Mirror Maker 2
Zookeeper
Cluster
Kafka Cluster
Kafka Connect
Cluster
Zookeeper
Cluster
Camel Kafka
Connectors
Debezium
Connectors
Java Clients /
Kafka Streams
(Inc Quarkus)
Manages
Manages
Deploys
Consumes
Validates
Validates
Replicates
Consumes / Produces
Consumes / Produces
Captures
Connects
Connects
Converts
Java Clients /
Kafka Streams
(Inc Quarkus)
API Management
Manages
Single Sign On
Authenticates

● Getting Started
● https://developers.redhat.com/blog/2019/12/16/getting-started-with-red-hat-integration-service-registry
● https://developers.redhat.com/blog/2020/12/09/new-features-and-storage-options-in-red-hat-integration-service-r
egistry-1-1-ga
● Install and configure Service Registry for AMQ Streams
● Installing Service Registry from the OpenShift OperatorHub
● Configuring Service Registry with AMQ Streams storage on OpenShift
● Validating Schemas using Kafka Client Serializer/Deserializer
● Validating schemas using Kafka client serializers/deserializers
● Strategies to look up a schema
● Service Registry Java Client
● Managing Service Registry content using the Java client
Useful Bookmarks
35

linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
36
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning
support, training, and consulting services make
Red Hat a trusted adviser to the Fortune 500.
Thank you

Available for different
solutions like Events (Kafka),
API management (3scale)
and EIPs (Fuse)
SERVICE REGISTRY
Service Registry
37
Targeted requirements for enterprise registry
Source:
Flexibility
Provide Lifecycle Management,
Monitoring & Metrics,
Operators, Pluggable Storage,
validations.
Enterprise Ready
Target usage for OpenAPI,
Async API, REST CRUD, UI,
Search
API Usage
Support different artifacts:
Avro, Protobuf, JSONSchema,
OpenAPI, AsyncAPI,
among others
Inclusive Artifacts

● Datastore for standard event schemas and API designs
● Handles following data formats:
○ Apache Avro
○ JSON Schema
○ Protobuf (protocol buffers)
○ OpenAPI
○ AsyncAPI
○ GraphQL
○ Kafka Connect Schema
● Drop in replacement for Confluent Schema Registry
SERVICE REGISTRY
Service Registry
38
Overview
Source:

● Server
● Java REST Client
○ Perform basic operations
● Utilities
○ Kafka Serializer / Deserializer (Serde)
○ Kafka Connect Converters
○ Maven Plugin
■ Register artifacts
■ Test rules and compatibility
■ Download artifacts
SERVICE REGISTRY
Service Registry
39
Components
Source:

● Java + Quarkus based implementation
● Pluggable Storage
○ Kafka Streams (GA), JPA (TP), Infinispan (TP)
● Apicurio
○ Built on the Apicurio Registry community project
○ API first design
○ Custom REST API
○ Confluent (and IBM) compliant API
SERVICE REGISTRY
Service Registry
40
Implementation
Source:

● Validation Rules
○ Artifact must have valid content or
server will reject it
○ Can check for valid syntax
○ Can also check for valid semantics
(for some artifact types)
SERVICE REGISTRY
Service Registry
41
● Compatibility Rules
○ Determines whether an update is allowed
based on configured compatibility
requirement setting
○ Multiple compatibility options including
Backwards and Forwards compatible
○ Only relevant for updates (checks the new
version against the previous version)
○ Controls the evolution of a single Artifact
over time
Registry Rules
Source:

● Backward Compatibility
○ New version of the artifact would be compatible with previous versions
of that artifact
○ Records that use the previous version can be read using the new
artifact
● Forward Compatibility
○ Exiting artifact is compatible with future versions of that artifact
○ Records that use the new artifact can still be retrieve using previous
version.
SERVICE REGISTRY
Service Registry
42
Compatibility Rules
Source:

● Byte format is more efficient in Kafka
● Multi protocol support (Avro, Protobuf, OpenAPI, JSON, …)
● Ensures consistent data format
○ A producer client application uses a serializer to put the messages
that it sends to a specific broker topic into the correct data format
○ A consumer client application uses a deserializer to get the messages
that it consumes from a specific broker topic into the correct data format
SERVICE REGISTRY
Client Serializer / Deserializer
43
Source:
https://access.redhat.com/documentation/en-us/red_hat_integration/2020-q4/html/getting_started_with_service_registry/using-kafka-client-serdes

Available for different
solutions like Events (Kafka),
API management (3scale)
and EIPs (Fuse)
APICURIO REGISTRY
Apicurio Registry
44
Overview
Source:
API designs registry
Provide Lifecycle Management,
Monitoring & Metrics,
Operators, Pluggable Storage,
validations.
Enterprise Ready
Target usage for OpenAPI,
Async API, REST CRUD, UI,
Search
API Usage
Support different artifacts:
Avro, Protobuf, JSONSchema,
OpenAPI, AsyncAPI,
among others
Schema Registry

Skeleton:
Intro to Api First: ( why contract first? )
- Opena api
- Async api
Schemas in event driven architectures
Schema registry for kafka applications
Apicurio Registry 2.0
- New v2 REST API
- New serdes
- CNCF Schema Registry API support
- Support for webhooks
Demo
- Basic config with spring boot?
- Kafka streams on quarkus
- Smallrye-reactive-messaging on quarkus
45

46
Event-driven APIs & schema governance
for Apache Kafka
Apicurio Registry
Fabian Martinez
Senior Software Quality Engineer
@famargon13
EVENT-DRIVEN
APIS
&
SCHEMA
GOVERNANCE
Hugo Guerrero
APIs & Messaging Developer Advocate
@hguerreroo

linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat
47
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning support,
training, and consulting services make Red Hat a trusted
adviser to the Fortune 500.
Thank you
EVENT-DRIVEN
APIS
&
SCHEMA
GOVERNANCE

48
What issues needs to be addressed in a Kafka world?
● Central registry where the API schemas (contracts) are stored
and accessible
● Kafka is not aware of data types or schema formats
> An instance to perform checks is needed
● API schemas are subject to change
> Version control, compatibility and validity needs to be governed

Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | Fabian Martinez and Hugo Guerrero, Red Hat

Recommended

More Related Content

What's hot (20)

Similar to Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | Fabian Martinez and Hugo Guerrero, Red Hat (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | Fabian Martinez and Hugo Guerrero, Red Hat