Confluent Operations Training for Apache Kafka

Apr 18, 20171 like748 views

This three-day hands-on course teaches students how to build, manage, and monitor Apache Kafka clusters using industry best practices. Students will learn how Kafka works, how to set up and manage a cluster, monitor performance, recover from failures, and secure the cluster. The course includes exercises on installing, operating, and tuning a Kafka cluster. It is designed for engineers and operations staff responsible for Kafka clusters.

© 2015-2017 Confluent, Inc D1_705www.confluent.io/contact
OPERATIONS TRAINING
Confluent Operations Training for Apache Kafka
Course Objectives
In this three-day hands-on course you will learn how to build, manage, and monitor clusters using industry best-practices developed
by the world’s foremost Apache KafkaTM
experts. You will learn how Kafka and the Confluent Platform work, their main subsystems,
how they interact, and how to set up, manage, monitor, and tune your cluster.
Hands-On Training
Throughout the course, hands-on
exercises reinforce the topics being
discussed. Exercises include:
»» Cluster installation
»» Basic cluster operations
»» Viewing and interpreting
cluster metrics
»» Recovering from a Broker failure
»» Performance-tuning the cluster
»» Securing the cluster
Who Should Attend?
This course is designed for
engineers, system administrators,
and operations staff responsible
for building, managing, monitoring,
and tuning Kafka clusters.
Course Duration
This is a three-day training course.
Course Prerequisites
Attendees should have a strong
knowledge of Linux/Unix, and
understand basic TCP/IP networking
concepts. Familiarity with the
Java Virtual Machine (JVM) is
helpful. Prior knowledge of Kafka
is helpful, but is not required.
Confluent offers public training courses and private, onsite events. Please visit http://confluent.io/training for the public course schedule.
For inquiries about onsite events, email training-admin@confluent.io
Introduction
The Motivation for Apache Kafka
• Systems Complexity
• Real-Time Processing is Becoming
Prevalent
• Kafka: A Stream Data Platform
Kafka Fundamentals
• An Overview of Kafka
• Kafka Producers
• Kafka Brokers
• Kafka Consumers
• Kafka’s Use of ZooKeeper
• Comparisons with Traditional
Message Queues
Providing Durability
• Basic Replication Concepts
• Durability Through Intra-Cluster
Replication
• Writing Data to Kafka Reliably
• Broker Shutdown and Failures
• Controllers in the Cluster
• The Kafka Log Files
• Offset Management
Designing for High Availability
• Kafka Reference Architecture
• Brokers
• ZooKeeper
• Connect
• Schema Registry
• REST Proxy
• Multiple Data Centers
Managing a Kafka Cluster
• Installing and Running Kafka
• Monitoring Kafka
• Basic Cluster Management
• Log Retention and Compaction
• An Elastic Cluster
Optimizing Kafka Performance
• Producer Performance
• Broker Performance
• Broker Failures and Recovery Time
• Load Balancing Consumption
• Consumption Performance
• Performance Testing
Kafka Security
• SSL for Encryption and
Authentication
• SASL for Authentication
• Data at Rest Encryption
• Securing ZooKeeper and the REST
Proxy
• Migration to a Secure Cluster
Integrating Systems with Kafka
Connect
• The Motivation for Kafka Connect
• Types of Connectors
• Kafka Connect Implementation
• Standalone and Distributed Modes
• Configuring the Connectors
• Deployment Considerations
• Comparison with Other Systems
Conclusion

This three-day course teaches developers how to build applications that can publish and subscribe to data from an Apache Kafka cluster. Students will learn Kafka concepts and components, how to use Kafka and Confluent APIs, and how to develop Kafka producers, consumers, and streams applications. The hands-on course covers using Kafka tools, writing producers and consumers, ingesting data with Kafka Connect, and more. It is designed for developers who need to interact with Kafka as a data source or destination.

How Apache Kafka® Worksconfluent

Watch this talk here: https://www.confluent.io/online-talks/how-apache-kafka-works-on-demand Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees. We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview. This session is part 3 of 4 in our Fundamentals for Apache Kafka series.

Give Your Confluent Platform Superpowers! (Sandeep Togrika, Intel and Bert Ha...HostedbyConfluent

Whether you are a die-hard DC comic enthusiast, mad for Marvel, or completely clueless when it comes to comic books, at the end of the day each of us would love to possess the superpower to transform data in seconds versus minutes or days. But architects and developers are challenged with designing and managing platforms that scale elastically and combine event streams with stored data, to enable more contextually rich data analytics. This made even more complex with data coming from hundreds of sources, and in hundreds of terabytes, or even petabytes, per day. Now, with Apache Kafka and Intel hardware technology advances, organizations can turn massive volumes of disparate data into actionable insights with the ability to filter, enrich, join and process data instream. Let's consider Information Security. IT leaders need to ensure all company data and IP is secured against threats and vulnerabilities. A combination of real-time event streaming with Confluent Platform and Intel Architecture has enabled threat detection efforts that once took hours to be completed in seconds, while simultaneously reducing technical debt and data processing and storage costs. In this session, Confluent and Intel architects will share detailed performance benchmarking results and new joint reference architecture. We’ll detail ways to remove Kafka performance bottlenecks, and improve platform resiliency and ensure high availability using Confluent Control Center and Multi-Region Clusters. And we’ll offer up tips for addressing challenges that you may be facing in your own super heroic efforts to design, deploy, and manage your organization’s data platforms.

Kafka Pluggable Authorization for Enterprise Security (Anna Kepler, Viasat) K...confluent

At Viasat, Kafka is a backbone for a multi-tenant streaming platform that transports data for 1000 streams and used by more than 60 teams in a production environment. Role-based access control to the sensitive data is an essential requirement for our customers who must comply with a variety of regulations including GDPR. Kafka ships with a pluggable Authorizer that can control access to resources like cluster, topic or consumer group. However, maintaining ACLs in the large multi-tenant deployment can be support-intensive. At Viasat, we developed a custom Kafka Authorizer and Role Manager application that integrates our Kafka cluster with Viasat’s internal LDAP services. The presentation will cover how we designed and built Kafka LDAP Authorizer, which allows us to control resources within the cluster as well as services built around Kafka. We apply our permissions model to our data forwarders, ETL jobs, and stream processing. We will also share how we achieved a stress free migration to secure infrastructure without interruption to the production data flow. Our secure deployment model accomplishes multiple goals: – Integration into an LDAP central authentication system. – Use of the same authorization service to control permissions to data in Kafka as well as services built around Kafka. – Delegation of permissions control to the security officers on the teams using the service. – Detailed audit and breach notifications based on the metrics produced by the custom authorizer. We plan to open source our custom Kafka Authorizer.

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...HostedbyConfluent

Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including: * How to manage multi-tenant clusters in a hybrid environment * High volume data pipelines with Mirus replicating data to Kafka and blob storage * Kafka Fault Injection Framework built on Trogdor and Kibosh * Automated recovery without data loss * Using Envoy as an SNI-routing Kafka gateway We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.

Understanding Apache Kafka® Latency at Scaleconfluent

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...HostedbyConfluent

You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.

Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, QlikHostedbyConfluent

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff ...HostedbyConfluent

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Red Hat Developers

This document discusses Apache Kafka and Red Hat OpenShift Streams for Apache Kafka. It begins with an overview of what Apache Kafka is and its common use cases. It then demonstrates how Red Hat OpenShift Streams provides a managed Apache Kafka cluster as a service, including a dedicated cluster, configuration management, metrics, monitoring and other features to provide a streamlined developer experience. It concludes with information on trying OpenShift Streams for Apache Kafka and additional resources.

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...HostedbyConfluent

Cloud is changing the world; Kubernetes is changing the world; real-time event streaming is changing the world. In this talk we explore some of best practices to synergistically combine the power of these paradigm shifts to achieve a much greater return on your Kafka investments. From declarative deployments, zero-downtime upgrades, elastic scaling to self-healing and automated governance, learn how you can bring the next level of speed, agility, resilience, and security to your Kafka implementations.

The best of Apache Kafka Architecturetechmaddy

Event Driven Architectures with Apache Kafka on HerokuHeroku

Apache Kafka is the backbone for building architectures that deal with billions of events a day. Chris Castle, Developer Advocate, will show you where it might fit in your roadmap. - What Apache Kafka is and how to use it on Heroku - How Kafka enables you to model your data as immutable streams of events, introducing greater parallelism into your applications - How you can use it to solve scale problems across your stack such as managing high throughput inbound events and building data pipelines Learn more at https://www.heroku.com/kafka Reveal.js version of slides: http://slides.com/christophercastle/deck#/

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...HostedbyConfluent

One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions. Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.

Everything you ever needed to know about Kafka on Kubernetes but were afraid ...HostedbyConfluent

Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.

Oops! I started a broker | Yinon Kahta, TaboolaHostedbyConfluent

What happened when our biggest and most important Kafka cluster went rogue all of a sudden, and while trying to recover it, a single, crucial misconfiguration made things even worse? At a company like Taboola, where service availability and latency are our top priority, this was a disaster. With 300K messages/sec and 250TB of messages produced each day to our on-premise Kafka clusters, and mirrored to our central Kafka cluster, we always try to ensure Kafka behaves well under high loads of traffic and unexpected cluster failures. So when our main Kafka cluster went crazy we had a serious issue on our hands. This session is the story of how we learned the hard way about mitigating cluster failures with the proper configurations in place.

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...confluent

This document discusses building microservices for data streaming and processing using Spring Cloud and Kafka. It provides an overview of Spring Cloud Stream and how it can be used to build event-driven microservices that connect to Kafka. It also discusses how Spring Cloud Data Flow can be used to orchestrate and deploy streaming applications and topologies. The document includes code samples of building a basic Kafka Streams processor application using Spring Cloud Stream and deploying it as part of a streaming data flow. It concludes with proposing a demonstration of these techniques.

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020confluent

Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us. Things I will be sharing as part of the talk: 1. Establishing the use case/problem we were trying to solve (why we needed mTLS) 2. Building a Certificate Authority with open source tools (with self-signed Root CA) 3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers 4. Hot reloading regenerated certificates on Kafka brokers without downtime 5. What we built to rotate the self-signed root CA without downtime as well across the board 6. Monitoring and alerts on TTL of certificates 7. Performance impact of using TLS (along with why TLS affects kafka’s performance) 8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier 9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...HostedbyConfluent

Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability. This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment: kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability. kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters. We'll also present challenges we encountered along the way and share our learnings with the audience.

Capacity Planning Your Kafka Cluster | Jason Bell, DigitalisHostedbyConfluent

"There's little talk about capacity planning Kafka clusters, it's very much learn as you go, every cluster is different. In this talk Kafka DevOps Engineer Jason Bell takes you through the things that will help you, from broker capacity, thinking about topics and how the other Confluent components can affect throughput and performance. With a number of production deployments under his watchful gaze for over six years Jason has plenty of experience, stories and useful information that will help you. By the end of the talk you'll have a good understanding of designing the cluster for various scenarios, where the points of latency are to watch and monitor. And also how to prevent teams breaking the cluster behind your back. This talk is designed for everyone, anyone who is just starting to those who are operating Kafka on a daily basis."

Leveraging Microservices and Apache Kafka to Scale Developer Productivityconfluent

Protecting your data at rest with Apache Kafka by Confluent and Vormetricconfluent

This document discusses securing Apache Kafka deployments with Vormetric and Confluent Platform. It begins with an introduction to Apache Kafka and Confluent Platform. It then provides an overview of Vormetric's policy-driven security solution and how it can be used to encrypt Kafka data at rest. The document outlines the typical Confluent Platform deployment architecture and various security considerations, such as authentication, authorization, and data encryption. Finally, it provides steps for implementing secure deployments using SSL, Kerberos, and Vormetric encryption policies.

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architectureconfluent

An Introduction to Confluent Cloud: Apache Kafka as a Serviceconfluent

Business breakout during Confluent’s streaming event in Munich, presented by Hans Jespersen, VP WW Systems Engineering at Confluent. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Westpac AU - Confluent Schema Registryconfluent

This document discusses using schema validation and a schema registry to ensure compatibility when data is serialized and transmitted between multiple applications and data sources. It introduces common challenges like different data formats and schemas causing issues. It then explains how tools like Avro schemas and a schema registry can define data contracts and validate messages to solve compatibility problems at scale. The document also considers how these tools integrate into Kafka pipelines and addresses questions around their usage across many developers and moving to the cloud.

Look how easy it is to go from events to blazing-fast analytics! | Neha Pawar...HostedbyConfluent

Apache Kafka is the de facto standard for real-time event streaming, but what do you do if you want to perform user-facing, ad-hoc, real-time analytics too? That's a hard problem. Apache Pinot solves it, and the two together are like chocolate and peanut butter, peaches and cream, and Steve Rogers and Peggy Carter. Come to this talk for an introduction to Pinot and an overview of how the Pinot Kafka Connector works. Hear the challenges unique to a user-facing realtime analytics system, and how Pinot and Kafka work harmoniously to solve them. Witness an action-packed demo, showing just how easy it is to go from events to blazing-fast analytics, and how to use powerful features of both systems that help you do this at scale.

Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...HostedbyConfluent

Whilst Kafka has the ability to encrypt data in transit, it does not have the functionality out of the box to encrypt data at rest. This places the responsibility of encryption of data placed on message queues on developers. Implementing cryptography correctly in our applications is challenging and time consuming. In this demo-driven talk, I will show you how you can use HashiCorp Vault’s API to implement a simple workflow that offsets the complexity of cryptography to Vault. In just a few lines of code, I will demonstrate how message producers will be able to encrypt its data, whilst message consumers can decrypt message payloads with minimal development effort. I will also show how to troubleshoot common errors from the API. By the end of this talk, you will learn how to implement symmetric and asymmetric encryption of your application data before placing it on Kafka message queues. You will also learn how to implement this workflow using Format Preserving Encryption (FPE).

Scaling an Event-Driven Architecture with IBM and Confluent | Antony Amanse a...HostedbyConfluent

New Features in Confluent Platform 6.0 / Apache Kafka 2.6Kai Wähner

Event Driven Architectures with Apache KafkaMatt Masuda

This document discusses event-driven architectures and how Apache Kafka can be used to enable them. It provides an overview of microservices architectures and the issues they can have with synchronous calls. Event-driven architectures address these issues using asynchronous messaging. Kafka is then introduced as a distributed messaging platform that allows publishing and subscribing to event streams. It describes key Kafka concepts like topics, partitions, producers, and consumers. The document argues that using Kafka for event-driven architectures solves problems around service location, load balancing, and integration of new services. It also provides durable storage and read positioning capabilities. Finally, it references additional resources and promises a demo.

More Related Content

What's hot (20)