Kafka and Avro with Confluent Schema Registry

™
Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka
Consulting
Avro
Kafka & Avro:
Confluent Schema
Registry
Managing Record Schema in
Kafka

Consulting
™
Confluent Schema Registry
❖ Confluent Schema Registry stores Avro Schemas for Kafka
clients
❖ Provides REST interface for putting and getting Avro schemas
❖ Stores a history of schemas
❖ versioned
❖ allows you to configure compatibility setting
❖ supports evolution of schemas
❖ Provides serializers used by Kafka clients which handles schema
storage and serialization of records using Avro

Consulting
™
Why Schema Registry?
❖ Producer creates a record/message, which is an Avro record
❖ Record contains the schema and data
❖ Schema Registry Avro Serializer serializes the data and schema id (just id)
❖ Keeps a cache of registered schemas from Schema Registry to ids
❖ Consumer receives payload and deserializes it with Schema Registry Avro Deserializers
❖ Deserializer looks up the full schema from cache or Schema Registry based on id
❖ Consumer has its schema, one it is expecting record/message to conform to
❖ Compatibility check is performed or two schemas
❖ if no match, but are compatible, then payload transformation happens aka Schema Evolution
❖ if not failure
❖ Kafka records have Key and Value and schema can be done on both

Consulting
™
Schema Compatibility
❖ Backward Compatibility (default)
❖ New, backward compatible schema, will not break consumers
❖ Producers could be using older schema that is backwards compatible with Consumer
❖ Forward compatibility
❖ Records sent with new forward compatible schema can be deserialized with older schemas
❖ Consumers can use an older schema and never be updated (maybe never needs new fields)
❖ Full compatibility
❖ New version of a schema is backward and forward compatible
❖ None
❖ Schema will not be validated for compatibility at all

Consulting
™
Schema Registry Config
❖ Compatibility can be configured globally or per schema
❖ Options are:
❖ NONE - don’t check for schema compatibility
❖ FORWARD - check to make sure last schema version is forward
compatible with new schemas
❖ BACKWARDS (default) - make sure new schema is backwards
compatible with latest
❖ FULL - make sure new schema is forwards and backwards
compatible from latest to new and from new to latest

Consulting
™
Schema Registry Actions
❖ Register schemas for key and values of Kafka records
❖ List schemas (subjects)
❖ List all versions of a subject (schema)
❖ Retrieve a schema by version or id
❖ get latest version of schema
❖ Check to see if schema is compatible with a certain version
❖ Get the compatibility level setting of the Schema Registry
❖ BACKWARDS, NONE
❖ Add compatibility settings to a subject/schema

Consulting
™
Schema Evolution
❖ Avro schema is changed after data has been written to store using an older version of
that schema, then Avro might do a Schema Evolution
❖ Schema evolution is automatic transformation of Avro schema
❖ transformation is between version of consumer schema and what the producer put
into the Kafka log
❖ When Consumer schema is not identical to the Producer schema used to serialize
the Kafka Record then a data transformation is performed on the Kafka record (key or
value)
❖ If the schemas match then no need to do a transformation
❖ Schema evolution is happens only during deserialization at the Consumer
❖ If Consumer’s schema is different from Producer’s schema, then value or key is
automatically modified during deserialization to conform to consumers reader schema

Consulting
™
Allowed Schema Modifications
❖ Add a field with a default
❖ Remove a field that had a default value
❖ Change a fields order attribute
❖ Change a fields default value
❖ Remove or add a field alias
❖ Remove or add a type alias
❖ Change a type to a union that contains original type

Consulting
™
Rules of the Road for modifying
Schema
❖ Provide a default value for fields in your schema
❖ Allows you to delete the field later later
❖ Don’t change a field's data type
❖ When adding a new field to your schema, you have to
provide a default value for the field
❖ Don’t rename an existing field
❖ You can add an alias

Consulting
™
Remember our example
Employee
Avro covered in Avro/Kafka Tutorial

Consulting
™
Let’s say
❖ Employee did not have an age in version 1 of the
schema
❖ Later we decided to add an age field with a default value
of -1
❖ Now let’s say we have a Producer using version 2, and
a Consumer using version 1

Consulting
™
Scenario adding a new field age with
default value
❖ Producer uses version 2 of the Employee schema and creates a
com.cloudurable.Employee record, and sets age field to 42, then sends it to Kafka topic
new-employees
❖ Consumer consumes records from new-employees using version 1 of the Employee
Schema
❖ Since Consumer is using version 1 of schema, age field is removed during
deserialization
❖ Same consumer modifies name field and then writes the record back to a NoSQL store
❖ When it does this, the age field is missing from value that it writes to the store
❖ Another client using version 2 reads the record from the NoSQL store
❖ Age field is missing from the record (because the Consumer wrote it with version 1),
age is set to default value of -1

Consulting
™
Schema Registry Actions
❖ Register schemas for key and values of Kafka records
❖ List schemas (subjects)
❖ List all versions of a subject (schema)
❖ Retrieve a schema by version or id
❖ get latest version of schema
❖ Check to see if schema is compatible with a certain version
❖ Get the compatibility level setting of the Schema Registry
❖ BACKWARDS, FORWARD, FULL, NONE
❖ Add compatibility settings to a subject/schema

Consulting
™
Register a Schema

Consulting
™
Register a Schema
{"id":2}
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json"
--data '{"schema": "{"type": …}’
http://localhost:8081/subjects/Employee/versions

Consulting
™
List All Schema
["Employee","Employee2","FooBar"]
curl -X GET http://localhost:8081/subjects

Consulting
™
Working with versions
[1,2,3,4,5]
{“subject”:"Employee","version":2,"id":4,"schema":"
{"type":"record","name":"Employee",
”namespace”:"com.cloudurable.phonebook", …
{“subject”:"Employee","version":1,"id":3,"schema":"
{"type":"record","name":"Employee",
”namespace”:"com.cloudurable.phonebook", …

Consulting
™
Working with Schemas

Consulting
™
Changing Compatibility
Checks

Consulting
™
Incompatible Change
{“error_code":409,"
message":"Schema being registered is incompatible with an e

Consulting
™
Incompatible Change
{"is_compatible":false}

Consulting
™
Use Schema Registry
❖ Start up Schema Registry server pointing to Zookeeper
cluster
❖ Import Kafka Avro Serializer and Avro Jars
❖ Configure Producer to use Schema Registry
❖ Use KafkaAvroSerializer from Producer
❖ Configure Consumer to use Schema Registry
❖ Use KafkaAvroDeserializer from Consumer

Consulting
™
Start up Schema Registry
Server
cat ~/tools/confluent-3.2.1/etc/schema-registry/schema-registry.properties
listeners=http://0.0.0.0:8081
kafkastore.connection.url=localhost:2181
kafkastore.topic=_schemas
debug=false

Consulting
™
Import Kafka Avro Serializer &
Avro Jars

Consulting
™
Configure Producer to use Schema
Registry

Consulting
™
Use KafkaAvroSerializer from
Producer

Consulting
™
Configure Consumer to use Schema
Registry

Consulting
™
Use KafkaAvroDeserializer from
Consumer

Consulting
™
Schema Registry
❖ Confluent provides Schema Registry to manage Avro
Schemas for Kafka Consumers and Producers
❖ Avro provides Schema Migration
❖ Confluent uses Schema compatibility checks to see if
Producer schema and Consumer schemas are
compatible and to do Schema evolution if needed
❖ Use KafkaAvroSerializer from Producer
❖ Use KafkaAvroDeserializer from Consumer

Kafka and Avro with Confluent Schema Registry

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Kafka and Avro with Confluent Schema Registry (20)

Recently uploaded (20)

Kafka and Avro with Confluent Schema Registry