Confluent Developer Training

Apr 18, 20170 likes803 views

This three-day course teaches developers how to build applications that can publish and subscribe to data from an Apache Kafka cluster. Students will learn Kafka concepts and components, how to use Kafka and Confluent APIs, and how to develop Kafka producers, consumers, and streams applications. The hands-on course covers using Kafka tools, writing producers and consumers, ingesting data with Kafka Connect, and more. It is designed for developers who need to interact with Kafka as a data source or destination.

This three-day hands-on course teaches students how to build, manage, and monitor Apache Kafka clusters using industry best practices. Students will learn how Kafka works, how to set up and manage a cluster, monitor performance, recover from failures, and secure the cluster. The course includes exercises on installing, operating, and tuning a Kafka cluster. It is designed for engineers and operations staff responsible for Kafka clusters.

Westpac AU - Confluent Schema Registryconfluent

This document discusses using schema validation and a schema registry to ensure compatibility when data is serialized and transmitted between multiple applications and data sources. It introduces common challenges like different data formats and schemas causing issues. It then explains how tools like Avro schemas and a schema registry can define data contracts and validate messages to solve compatibility problems at scale. The document also considers how these tools integrate into Kafka pipelines and addresses questions around their usage across many developers and moving to the cloud.

Leveraging Microservices and Apache Kafka to Scale Developer Productivityconfluent

Introduction to Apache Kafka and why it matters - MadridPaolo Castagna

This document provides an introduction to Apache Kafka and discusses why it is an important distributed streaming platform. It outlines how Kafka can be used to handle streaming data flows in a reliable and scalable way. It also describes the various Apache Kafka APIs including Kafka Connect, Streams API, and KSQL that allow organizations to integrate Kafka with other systems and build stream processing applications.

Give Your Confluent Platform Superpowers! (Sandeep Togrika, Intel and Bert Ha...HostedbyConfluent

Whether you are a die-hard DC comic enthusiast, mad for Marvel, or completely clueless when it comes to comic books, at the end of the day each of us would love to possess the superpower to transform data in seconds versus minutes or days. But architects and developers are challenged with designing and managing platforms that scale elastically and combine event streams with stored data, to enable more contextually rich data analytics. This made even more complex with data coming from hundreds of sources, and in hundreds of terabytes, or even petabytes, per day. Now, with Apache Kafka and Intel hardware technology advances, organizations can turn massive volumes of disparate data into actionable insights with the ability to filter, enrich, join and process data instream. Let's consider Information Security. IT leaders need to ensure all company data and IP is secured against threats and vulnerabilities. A combination of real-time event streaming with Confluent Platform and Intel Architecture has enabled threat detection efforts that once took hours to be completed in seconds, while simultaneously reducing technical debt and data processing and storage costs. In this session, Confluent and Intel architects will share detailed performance benchmarking results and new joint reference architecture. We’ll detail ways to remove Kafka performance bottlenecks, and improve platform resiliency and ensure high availability using Confluent Control Center and Multi-Region Clusters. And we’ll offer up tips for addressing challenges that you may be facing in your own super heroic efforts to design, deploy, and manage your organization’s data platforms.

Event Driven Architectures with Apache Kafka on HerokuHeroku

Apache Kafka is the backbone for building architectures that deal with billions of events a day. Chris Castle, Developer Advocate, will show you where it might fit in your roadmap. - What Apache Kafka is and how to use it on Heroku - How Kafka enables you to model your data as immutable streams of events, introducing greater parallelism into your applications - How you can use it to solve scale problems across your stack such as managing high throughput inbound events and building data pipelines Learn more at https://www.heroku.com/kafka Reveal.js version of slides: http://slides.com/christophercastle/deck#/

Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드confluent

Intro to AsyncAPIconfluent

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...confluent

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Red Hat Developers

This document discusses Apache Kafka and Red Hat OpenShift Streams for Apache Kafka. It begins with an overview of what Apache Kafka is and its common use cases. It then demonstrates how Red Hat OpenShift Streams provides a managed Apache Kafka cluster as a service, including a dedicated cluster, configuration management, metrics, monitoring and other features to provide a streamlined developer experience. It concludes with information on trying OpenShift Streams for Apache Kafka and additional resources.

Introducing Confluent Cloud: Apache Kafka as a Service confluent

1. The document introduces Confluent Cloud, which provides Apache Kafka as a service and allows organizations to liberate developers from managing Kafka infrastructure. 2. Confluent Cloud is built on open source Apache Kafka and provides access to the rich Kafka ecosystem through pre-built connectors while minimizing operations burden by handling infrastructure setup, cluster tuning, upgrades and more. 3. Key benefits of Confluent Cloud include freedom to deploy Kafka in any public cloud, leveraging the leading streaming platform to develop faster applications, and being relieved of operations management through a fully managed service.

An Introduction to Confluent Cloud: Apache Kafka as a Serviceconfluent

Business breakout during Confluent’s streaming event in Munich, presented by Hans Jespersen, VP WW Systems Engineering at Confluent. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...HostedbyConfluent

You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.

Twitter’s Apache Kafka Adoption Journey | Ming Liu, TwitterHostedbyConfluent

Until recently, the Messaging team at Twitter had been running an in-house build Pub/Sub system, namely EventBus (built on top of Apache DistributedLog and Apache Bookkeeper, and similar in architecture to Apache Pulsar) to cater to our pubsub needs. In 2018, we made the decision to move to Apache Kafka by migrating existing use cases as well as onboarding new use cases directly onto Apache Kafka. Fast forward to today, Kafka is now an essential piece of Twitter Infrastructure and processes over 200M messages per second. In this talk, we will share the learning and challenges in our journey moving to Apache Kafka.

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...confluent

What do you do when you've two different technologies on the upstream and the downstream that are both rapidly being adopted industrywide? How do you bridge them scalably and robustly? At Wework, the upstream data was being brokered by Kafka and the downstream consumers were highly scalable gRPC services. While Kafka was capable of efficiently channeling incoming events in near real-time from a variety of sensors that were used in select Wework spaces, the downstream gRPC services that were user-facing were exceptionally good at serving requests in a concurrent and robust manner. This was a formidable combination, if only there was a way to effectively bridge these two in an optimized way. Luckily, sink Connectors came to the rescue. However, there weren't any for gRPC sinks! So we wrote one. In this talk, we will briefly focus on the advantages of using Connectors, creating new Connectors, and specifically spend time on gRPC sink Connector and its impact on Wework's data pipeline.

How did we move the mountain? - Migrating 1 trillion+ messages per day across...HostedbyConfluent

Have you ever migrated Kafka clusters from one data center to another being completely transparent to client applications? At PayPal, as part of a massive datacenter migration initiative, Kafka team successfully moved all PayPal Kafka traffic across data centers. This initiative involved migrating 20+ Kafka clusters (1000+ broker and zookeeper nodes), as well as 60+ mirrormaker groups which seamlessly handle Kafka traffic volumes as high as 1 trillion messages per day. Throughout the course of this migration, applications required no modification, encountered 0% service outage, 0% message loss and duplicated messages. The whole migration process was fully transparent to Kafka applications. In this session, you will learn the strategies, techniques and tools the PayPal Kafka team has utilized for managing the migration process. You will also learn the lessons and pitfalls they experienced during this exercise, as well as the secret sauce of making the migration successful.

Introduction to Kafka connectKnoldus Inc.

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...HostedbyConfluent

When choosing an event streaming platform, Kafka shouldn’t be the only technology you look at. There are a plethora of others in the messaging space today, including open source and proprietary software as well as a range of cloud services. So how do you know you are choosing the right one? A great way to deepen our understanding of event streaming and Kafka is exploring the trade-offs in distributed system design and learning about the choices made by the Kafka project. We’ll look at how Kafka stacks up against other technologies in the space, including traditional messaging systems like Apache ActiveMQ and RabbitMQ as well as more contemporary ones, such as BookKeeper derivatives like Apache Pulsar or Pravega. This talk focuses on the technical details such as difference in messaging models, how data is stored locally as well as across machines in a cluster, when (not) to add tiers to your system, and more. By the end of the talk, you should have a good high-level understanding of how these systems compare and which you should choose for different types of use cases.

Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, QlikHostedbyConfluent

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

Kafka Streams for Java enthusiastsSlim Baltagi

Kafka, Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform (Kafka Core + Kafka Connect + Kafka Streams) for building streaming data pipelines and streaming data applications. This talk, that I gave at the Chicago Java Users Group (CJUG) on June 8th 2017, is mainly focusing on Kafka Streams, a lightweight open source Java library for building stream processing applications on top of Kafka using Kafka topics as input/output. You will learn more about the following: 1. Apache Kafka: a Streaming Data Platform 2. Overview of Kafka Streams: Before Kafka Streams? What is Kafka Streams? Why Kafka Streams? What are Kafka Streams key concepts? Kafka Streams APIs and code examples? 3. Writing, deploying and running your first Kafka Streams application 4. Code and Demo of an end-to-end Kafka-based Streaming Data Application 5. Where to go from here?

Making Kafka Cloud Native | Jay Kreps, Co-Founder & CEO, ConfluentHostedbyConfluent

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...confluent

This document discusses building microservices for data streaming and processing using Spring Cloud and Kafka. It provides an overview of Spring Cloud Stream and how it can be used to build event-driven microservices that connect to Kafka. It also discusses how Spring Cloud Data Flow can be used to orchestrate and deploy streaming applications and topologies. The document includes code samples of building a basic Kafka Streams processor application using Spring Cloud Stream and deploying it as part of a streaming data flow. It concludes with proposing a demonstration of these techniques.

Building Microservices with Apache Kafkaconfluent

Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics. Presentation by Colin McCabe, Confluent, Big Data Day LA

Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...HostedbyConfluent

Tips & Tricks for Apache Kafka®confluent

Kat Grigg, Confluent, Senior Customer Success Architect + Jen Snipes, Confluent, Senior Customer Success Architect This presentation will cover tips and best practices for Apache Kafka. In this talk, we will be covering the basic internals of Kafka and how these components integrate together including brokers, topics, partitions, consumers and producers, replication, and Zookeeper. We will be talking about the major categories of operations you need to be setting up and monitoring including configuration, deployment, maintenance, monitoring and then debugging. https://www.meetup.com/KafkaBayArea/events/270915296/

Streaming all over the world Real life use cases with Kafka Streamsconfluent

This document discusses using Apache Kafka Streams for stream processing. It begins with an overview of Apache Kafka and Kafka Streams. It then presents several real-life use cases that have been implemented with Kafka Streams, including data conversions from XML to Avro, stream-table joins for event propagation, duplicate elimination, and detecting absence of events. The document concludes with recommendations for developing and operating Kafka Streams applications.

Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...HostedbyConfluent

Having started with classic monolith applications in the late 90s and adopting a new microservice architecture in 2015, our organization needed a convenient, reliable, and low-cost way to push changes back and forth between them. One that preferably utilized technology already on hand and could exchange information between multiple data stores. In this session we will explore how Kafka Connect and its various connectors satisfied this need. We will review the two disparate tech stacks we needed to integrate, and the strategies and connectors we used to exchange information. Finally, we will cover some enhancements we made to our own processes including integrating Kafka Connect and its connectors into our CI/CD pipeline and writing tools to monitor connectors in our production environment.

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020confluent

Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us. Things I will be sharing as part of the talk: 1. Establishing the use case/problem we were trying to solve (why we needed mTLS) 2. Building a Certificate Authority with open source tools (with self-signed Root CA) 3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers 4. Hot reloading regenerated certificates on Kafka brokers without downtime 5. What we built to rotate the self-signed root CA without downtime as well across the board 6. Monitoring and alerts on TTL of certificates 7. Performance impact of using TLS (along with why TLS affects kafka’s performance) 8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier 9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)

Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...confluent

Event-based stream processing is a modern paradigm to continuously process incoming data feeds, e.g. for IoT sensor analytics, payment and fraud detection, or logistics. Machine Learning / Deep Learning models can be leveraged in different ways to do predictions and improve the business processes. Either analytic models are deployed natively in the application or they are hosted in a remote model server. In the latter you combine stream processing with RPC / Request-Response paradigm instead of direct doing direct inference within the application. This talk discusses the pros and cons of both approaches and shows examples of stream processing vs. RPC model serving using Kubernetes, Apache Kafka, Kafka Streams, gRPC and TensorFlow Serving. The trade-offs of using a public cloud service like AWS or GCP for model deployment are also discussed and compared to local hosting for offline predictions directly "at the edge".

Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...Kai Wähner

Talk from Kafka Summit San Francisco 2019 (https://kafka-summit.org/sessions/event-driven-model-serving-stream-processing-vs-rpc-kafka-tensorflow/). Video recording will be available for free on the Summit website. Event-based stream processing is a modern paradigm to continuously process incoming data feeds, e.g. for IoT sensor analytics, payment and fraud detection, or logistics. Machine Learning / Deep Learning models can be leveraged in different ways to do predictions and improve the business processes. Either analytic models are deployed natively in the application or they are hosted in a remote model server. In the latter you combine stream processing with RPC / Request-Response paradigm instead of direct doing direct inference within the application. This talk discusses the pros and cons of both approaches and shows examples of stream processing vs. RPC model serving using Kubernetes, Apache Kafka, Kafka Streams, gRPC and TensorFlow Serving. The trade-offs of using a public cloud service like AWS or GCP for model deployment are also discussed and compared to local hosting for offline predictions directly “at the edge”. Key takeaways • Machine Learning / Deep Learning models can be used in different ways to do predictions. Scalability and loose coupling are important success factors • Stream processing vs. RPC / Request-Response for model serving has many trade-offs – learn about alternatives and best practices for your different scenarios • Understand the alternatives and trade-offs of model deployment in modern infrastructures like Kubernetes or Cloud Services like AWS or GCP • See live demos with Java, gRPC, Apache Kafka, KSQL and TensorFlow Serving to understand the trade-offs

More Related Content

What's hot (20)