Explaining How Real-Time GenAI Works in a Noisy Pub

Apr 2, 20240 likes93 views

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining how GenAI works
( in a noisy pub )
Simon Aubury
@SimonAubury

Training
Pre training & ﬁne tuning
Large language models - 3 things …
Data
A bunch of text
Architecture
The transformer
3

5
“Beer”
Image by Republica from Pixabay
WebVectors Online for "beer"

Image by Christian_Birkholz from Pixabay

Understanding language can be challenging
9
The group of friends bought cold beer at the
local pub to celebrate their reunion.
context
context

Transformers
10
Source: Attention Is All You Need https://arxiv.org/abs/1706.03762

Self-attention
11
Inspired by DeepLearning.AI
The
group
of
friends
bought
cold
beer
The
group
of
friends
bought
cold
beer

A transformer + data
= a base model
12
The base model
Data
Transformer
Base model

Completion
Base model
13
Completions
Beer
is
served
best
cold
with
friends
Context
Base model + context
= completion

15
How to train your LLM
Pretraining (once)
1. PB’s of text.
2. 1000’s of GPUs.
3. Compress the text into
a neural network,
4. Pay 💰 wait 📆
➡ Obtain base model.
Fine-tuning (recurring)
1. 1000’s ideal Q&A
responses (human)
2. Finetune base model
on this data wait 📆
3. Obtain assistant model
4. Evaluate, deploy &
monitor

󰜄 It is faster to pick than to generate.
16
Reinforcement Learning from Human Feedback
(RLHF)
��
��

Training
Pre training & ﬁne tuning
Large language models
Data
A bunch of text
Architecture
The transformer
17

18
Retrieval Augmented Generation (RAG)
What drink should
I order?

“ The entire history of
software engineering
is one of rising levels
of abstraction.
Grady Booch, IBM chief scientist
19

20
Thanks!
@SimonAubury
❤Presentation template by SlidesCarnival, Icons Flaticon photos Unsplash and Pixabay

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Automating Speed: A Proven Approach to Preventing Performance Regressions in ...HostedbyConfluent

"Regular performance testing is one of the pillars of Kafka Streams’ reliability and efficiency. Beyond ensuring dependable releases, regular performance testing supports engineers in new feature development with the ability to easily test the performance impact of their features, compare different approaches, etc. In this session, Alex and John share their experience from developing, using, and maintaining a performance testing framework for Kafka Streams that has prevented multiple performance regressions over the last 5 years. They cover guiding principles and architecture, how to ensure statistical significance and stability of results, and how to automate regression detection for actionable notifications. This talk sheds light on how Apache Kafka is able to foster a vibrant open-source community while maintaining a high performance bar across many years and releases. It also empowers performance-minded engineers to avoid common pitfalls and bring high-quality performance testing to their own systems."

How to Build an Event-based Control Center for the Electrical GridHostedbyConfluent

"The energy transition has brought with it a wave of new challenges for the high-voltage grid. One of the central problems is the volatility of renewable energies. A future-proof control center must have enough flexibility to meet the demands of the energy market in real time. Existing industrial systems cannot meet the requirements of modern network operators. Our projects aim to solve these challenges with sophisticated event-based architectures, with Apache Kafka at its core. These architectures need to solve complex requirements like switching high critical grid assets and integrate hundreds of thousands of sensors. We used a couple of event- based patterns like SAGAs, Event Sourcing and more to overcome those challenges. Additionally, we had to integrate the OT systems in this modern tech stack. (IEC, OPCUA) Some projects are already in shadow operation and are expected to steer first grid parts in 2025."

Keep Your Kafka Cloud Costs in Check with ShowbacksHostedbyConfluent

"Apache Kafka is the cool kid on the block when it comes to distributed streaming platforms. It's gained popularity in cloud environments due to its scalability, fault tolerance, and real-time data processing capabilities. As more organizations move their data processing and analysis workflows to the cloud, Kafka has become an essential part of their architecture. But, let's face it, running Kafka in the cloud can be a bit of a headache. It requires careful planning and execution to ensure optimal performance, reliability, and cost-effectiveness. In this talk, we're going to introduce you to a little something called ""showbacks"". Showbacks are a way of giving users feedback on the resources they're consuming and the associated costs. It's like telling your friends to keep their hands out of the cookie jar before they eat all the cookies. It also works with developers and partitions. We'll even give you some real-world examples of how New Relic has successfully implemented showbacks to achieve cost savings and operational efficiency in our Kafka deployments like auto-scaling, Availability Zone awareness, or client tunning. We promise to keep it fun, technical, and full of practical insights and actionable recommendations to help you achieve optimal Kafka performance and cost-effectiveness in the cloud."

When Securing Access to Data is About Life and DeathHostedbyConfluent

"The Norwegian health care sectors success is dependent on the security and availability of enormous amounts of critical data. The state of affairs is that data does not flow. At Norsk helsenett we have recently started using Kafka to secure this flow. Our critical data is your critical data, and a matter of life and death. This I how we do that: - Redundancy to us starts with the number 3, creating the infrastructure and using it to manage ""always available"" services and data, working towards zero downtime and data loss. - Security is paramount, how did we implement mTLS? - How do we monitor and support usage of data we cannot look at using the benefits of .Net and SignalR in Blazor together with Kafka."

Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiHostedbyConfluent

"Invidi ad decisioning engine needs semi-realtime feedback on the performance of the ad campaigns it runs. In the heart of this feedback loop there is a service that aggregates 1B+ daily ad tracking events and serves campaign performance time series to the ad decisioning engine over http. Recently we successfully rewrote it as a pure Kafka Streams application with all data being stored in Kafka and served via Interactive Queries. The experience was surprisingly not straightforward and we had to trade off some of the simplicity of our processing topology to increase scalability and lower resource consumption. In this talk we plan to go over the system architecture and share the issues we faced and how we solved them. Here are some highlights: - The distribution of our aggregation keys is very skewed, so early repartitioning resulted in poor scalability. To mitigate this issue we used a scatter-gather approach avoiding repartitioning and combining results in IQ. To minimize memory consumption we had to combine the above approach with pre-aggregating events before re-partitioning in a lambda-architecture style. - Due to multiple stores sharing buffer memory we had to resort to manually deleting entries in our live windowed stores to avoid premature flushing of the aggregates due to cache thrashing. - We had to implement our own in-memory windowed store to increase IQ performance We hope that our findings will be helpful to a wider audience. We also plan to file and fix the issues we discovered in the near future."

Mastering Kafka Consumer Distribution: A Guide to Efficient Scaling and Resou...HostedbyConfluent

"Consumer scaling is a crucial element for many Apache Kafka users. Who doesn’t want to save money by efficiently managing their resources, shutting down unnecessary instances when there is no traffic, quickly scaling up during peak hours and while doing all of that - avoiding annoying and often unnecessary rebalancing. To achieve all of this you need to understand how consumer assignment works, how nodes are affected by data load and what are common causes of rebalancing. But most importantly - what assignors to choose based on your use case and what metrics to use to measure your data load. You wonder how we know what are good and bad practices? At Aiven, we've seen firsthand both successful and not-so-great approaches to consumer scaling and rebalancing. The insights we're sharing with you come directly from our experience working on many projects with Apache Kafka. We’ll discuss metrics that are essential for understanding data load and deciding when to scale. We'll cover a variety of approaches you can take - from commonly used lag exporters, to Knative scalers that are based on concurrent requests and finally insights from our own experience developing a speed lag predictor that goes beyond the basics by calculating the velocity of data load changes. We’ll highlight advantages and disadvantages of each approach and when you should use it. Next, we'll look at various assignors that are available and guide you on how to choose the most suitable one for your scenario. We'll pay special attention to the challenges faced by stateful applications and the potential pitfalls of frequent scaling, such as overloaded brokers. Armed with this knowledge, you’ll have what is needed to build scalable systems, minimize downtime and save costs when working with Apache Kafka. Let's make your Kafka experience as smooth and efficient as possible!"

Flink 2.0: Navigating the Future of Unified Stream and Batch ProcessingHostedbyConfluent

"The Apache Flink community is working on a significant milestone with the planned release of Flink 2.0, marking a major evolution since the inception of Flink 1.0 in 2016. In this insightful talk, we will delve into the key enhancements and transformations slated for the 2.0 version, offering a comprehensive overview for users and developers eager to embrace the next frontier of stream and batch processing. The talk will commence by exploring some of the core philosophies of Flink, emphasizing the unification of batch and streaming processing. We'll dissect the roadmap's commitment to a seamless blending of batch and streaming applications. Flink 2.0 isn't just about features; it's also a reimagining of Flink's architecture. We'll delve into the disaggregated state management approach, leveraging distributed file systems. This evolution is geared towards better load balancing and improved efficiency in cloud-native architectures. APIs play a pivotal role in Flink's evolution, and Flink 2.0 is no exception. The talk will outline plans to retire deprecated APIs, overhaul the configuration layer, new abstractions and of course other plans that live in the community. Join us on this exploration of Flink 2.0, where we'll unravel the future of stream and batch processing and showcase how these advancements will shape the landscape of real-time data analytics."

Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Stream...HostedbyConfluent

"In the realm of real time streaming application, Kafka is commonly chosen as the datastore. This demands the use of faster and expensive storage at scale. However, the need for high-performance storage volume often translates to steep costs, hindering its wider adoption. To tackle this challenge, we have integrated the tiered storage feature introduced in KIP-405 in our Strimzi-operated Kafka. With tiered storage and our custom remote plugin, we provide a cost-effective solution allowing data to be stored in a cheaper storage, and thus enabling extended retention periods and allow efficiently backfilling of past data. This innovation allows our streaming applications to benefit without the strain of hefty storage expenses. In this session, we'll familiarize audience on how we use Kafka to build an end-to-end streaming applications while also touching the behind-the-scene aspect of how we were able to reduce the cost to make this affordable for wider adoption. We will share our experience on integrating the tiered storage feature with Strimzi Kafka operator and Strimzi operated Kafka clusters. Moreover, we will share our journey of optimizing performance for our remote storage manager implementation and the valuable insights gained along the way."

Building Kafka Connectors with Kotlin: A Step-by-Step Guide to Creation and D...HostedbyConfluent

"Kafka Connect, the framework for building scalable and reliable data pipelines, has gained immense popularity in the data engineering landscape. This session will provide a comprehensive guide to creating Kafka connectors using Kotlin, a language known for its conciseness and expressiveness. In this session, we will explore a step-by-step approach to crafting Kafka connectors with Kotlin, from inception to deployment using an simple use case. The process includes the following key aspects: Understanding Kafka Connect: We'll start with an overview of Kafka Connect and its architecture, emphasizing its importance in real-time data integration and streaming. Connector Design: Delve into the design principles that govern connector creation. Learn how to choose between source and sink connectors and identify the data format that suits your use case. Building a Source Connector: We'll start with building a Kafka source connector, exploring key considerations, such as data transformations, serialization, deserialization, error handling and delivery guarantees. You will see how Kotlin's concise syntax and type safety can simplify the implementation. Testing: Learn how to rigorously test your connector to ensure its reliability and robustness, utilizing best practices for testing in Kotlin. Connector Deployment: go through the process of deploying your connector in a Kafka Connect cluster, and discuss strategies for monitoring and scaling. Real-World Use Cases: Explore real-world examples of Kafka connectors built with Kotlin. By the end of this session, you will have a solid foundation for creating and deploying Kafka connectors using Kotlin, equipped with practical knowledge and insights to make your data integration processes more efficient and reliable. Whether you are a seasoned developer or new to Kafka Connect, this guide will help you harness the power of Kafka and Kotlin for seamless data flow in your applications."

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

More Related Content

More from HostedbyConfluent (20)