Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020

Sep 10, 20200 likes1,296 views

SIEM platforms are essential to the new cybersecurity paradigm and data collection layer is a very important piece of it. When you deliver a new platform, you can easily get lost in a variety of different vendors and solutions, too many challenges are facing. What if I change vendors, will I keep my data? How to feed multiple tools with the same data? How to collect data from custom apps and services? How to pay less for an expensive platform? How to keep data without a huge cost? Join us if you are looking for the answers. In this session, you will learn how we replaced the vendor-provided data collection layer with kafka connect and the lessons we learnt. After the talk you will know: - architecture and real-life examples of the flexible and highly available data collection platform - custom connectors that do most of the work for us and how to extend the connectors to consume new data, we made them open sourced - easy way to receive data from thousands of servers and many cloud services - how to archive data at low cost You will leave armed with a set of free tools and recipes to build a truly vendor-agnostic data collection platform. It will allow you to take you SIEM costs under control. You will feed your analytics tools with what they need and archive the rest at low cost. You will feed your SIEM smart!

Feed Your SIEM Smart
with Kafka Connect
Vitalii Rudenskyi
Security Architect, McKesson Corp

Why We Started
It’s not a mistake
to make a mistake,
but it’s a mistake
to repeat the
same mistake

Requirements
★ Logs collection to be vendor agnostic
★ Feed different analytics tools
★ Data filtering and cleaning
★ Existing data retention requirements
★ Scalable and highly available

Anticipated Challenges
★ Amount of data and number of sources
★ Variety of different formats
★ SaaS/PaaS applications and public Clouds
★ Cloud SIEM

The Three Keys
‘Push’ Connector ‘Pull’ Connector
Transformations Library

Push - NettySource Connector
★ Unified: “all-in-one” different transport and protocols
★ Scalable: multiple tasks supported
★ Available: works behind a LB (health checks supported)
★ Configurable: multiple protocols (plain text, syslog, http, snmp,
netflow)
★ Customizable: custom implementations supported

NettySource Connector
Common deployment model

Pull - PollableAPIClient Connector
★ Simple: easy new connectors development
★ Scalable: multiple tasks/partitions supported
★ Conﬁgurable: interval and scheduled (cron-style) polls,
retry/backoff, resettable offsets
★ Customizable: custom implementations supported

PollableAPIClient Connector
25+ ApiClient implementations

Transformations Library
★ Can transform different parts of a kafka message
★ Supports “if” conditions

Headers All The Way
★ Track origin of the data
★ Conditional routing
★ Tagging in ‘chained’ transformations
★ SIEM specific metadata

Syslog: Take Port 514 Under Control
Multi-rules RegexRouter config

Data archiving solution
★ Pair of source and sink connectors
★ One connector for all topics
★ Compressed
★ Easy to restore

★ Monitoring is essential
★ Keep original data
★ Be cautious when using “heavy” transformations in
source connectors

5 kafka clusters
20+ kafka connect clusters
530+ deployed connectors
7+TB of data daily

We share!
★ NettySource Connector
https://github.com/vrudenskyi/kafka-connect-netty-source
★ PollableAPIClient Connector
https://github.com/vrudenskyi/kafka-connect-pollable-source
https://github.com/vrudenskyi/kafka-connect-api-clients
★ Transformations Library
https://github.com/vrudenskyi/kafka-connect-transform

Q & Avrudenskyi@gmail.com
https://www.linkedin.com/in/vrudenskyi/

This talk will teach you how to redesign an event-driven autoscaling architecture for cloud-native microservices by utilizing Apache Kafka, Knative, and KEDA infrastructure. You will also learn how to deploy serverless applications (Quarkus) using a Knative service. Finally, KEDA will enable you to autoscale Knative Eventing components (KafkaSource) through events consumption over standard resources (CPU, memory).

Designing a complete ci cd pipeline using argo events, workflow and cd productsJulian Mazzitelli

https://www.youtube.com/watch?v=YmIAatr3Who Presented at Cloud and AI DevFest GDG Montreal on September 27, 2019. Are you looking to get more flexibility out of your CICD platform? Interested how GitOps fits into the mix? Learn how Argo CD, Workflows, and Events can be combined to craft custom CICD flows. All while staying Kubernetes native, enabling you to leverage existing observability tooling.

OpenTelemetry For ArchitectsKevin Brockhoff

Building an open data platform with apache icebergAlluxio, Inc.

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent

Apache Hudi is a data lake platform, that provides streaming primitives (upserts/deletes/change streams) on top of data lake storage. Hudi powers very large data lakes at Uber, Robinhood and other companies, while being pre-installed on four major cloud platforms. Hudi supports exactly-once, near real-time data ingestion from Apache Kafka to cloud storage, which is typically used in-place of a S3/HDFS sink connector to gain transactions and mutability. While this approach is scalable and battle-tested, it can only ingest data in mini batches, leading to lower data freshness. In this talk, we introduce a Kafka Connect Sink Connector for Apache Hudi, which writes data straight into Hudi's log format, making the data immediately queryable, while Hudi's table services like indexing, compaction, clustering work behind the scenes, to further re-organize for better query performance.

Orion Context Broker 1.15.0Fermin Galan

The document provides an introduction to context management and the Orion Context Broker. It discusses how context information from various sources can be gathered and managed to enable smart applications. The Orion Context Broker implements the NGSI standard API for managing context information and integrates with existing systems through a common context information hub. It allows applications to easily get, update, and subscribe to context data by reading and changing attribute values of entities.

Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Flink Forward

Flink Forward San Francisco 2022. Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way. by Jeff Chao

Presto, Zeppelin을 이용한 초간단 BI 구축 사례Hyoungjun Kim

RedisConf18 - Redis on Google Cloud PlatformRedis Labs

This document provides an overview of Redis on Google Cloud Platform and their new fully managed Redis service called Cloud Memorystore for Redis. The key points are: - Google Cloud Platform offers a new managed Redis service called Cloud Memorystore for Redis that makes Redis fast, scalable, highly available, secure and fully managed. - Cloud Memorystore for Redis offers different tiers (Basic and Standard) with different availability levels and SLAs. It allows scaling instances seamlessly to achieve high throughput and low latency. - Using Cloud Memorystore for Redis provides increased reliability, security and ease of use compared to self-managed Redis, as Google handles the infrastructure management and maintenance.

Apache Iceberg: An Architectural Look Under the CoversScyllaDB

Data Lakes have been built with a desire to democratize data - to allow more and more people, tools, and applications to make use of data. A key capability needed to achieve it is hiding the complexity of underlying data structures and physical data storage from users. The de-facto standard has been the Hive table format addresses some of these problems but falls short at data, user, and application scale. So what is the answer? Apache Iceberg. Apache Iceberg table format is now in use and contributed to by many leading tech companies like Netflix, Apple, Airbnb, LinkedIn, Dremio, Expedia, and AWS. Watch Alex Merced, Developer Advocate at Dremio, as he describes the open architecture and performance-oriented capabilities of Apache Iceberg. You will learn: • The issues that arise when using the Hive table format at scale, and why we need a new table format • How a straightforward, elegant change in table format structure has enormous positive effects • The underlying architecture of an Apache Iceberg table, how a query against an Iceberg table works, and how the table’s underlying structure changes as CRUD operations are done on it • The resulting benefits of this architectural design

KSQL - Stream Processing simplified!Guido Schmutz

KSQL is a stream processing SQL engine, which allows stream processing on top of Apache Kafka. KSQL is based on Kafka Stream and provides capabilities for consuming messages from Kafka, analysing these messages in near-realtime with a SQL like language and produce results again to a Kafka topic. By that, no single line of Java code has to be written and you can reuse your SQL knowhow. This lowers the bar for starting with stream processing significantly. KSQL offers powerful capabilities of stream processing, such as joins, aggregations, time windows and support for event time. In this talk I will present how KSQL integrates with the Kafka ecosystem and demonstrate how easy it is to implement a solution using KSQL for most part. This will be done in a live demo on a fictitious IoT sample.

Elk Caleb Wang

Elk - An introductionHossein Shemshadi

So, what is the ELK Stack? "ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...DataWorks Summit

Ranger’s pluggable architecture allows resource access policy administration and enforcement for standard and custom services from a “single pane of glass”. Apache Ranger has a rich Authorization Model, which provides the mechanism to author Policy in a Ranger Admin Server and serves as policy decision and audit point in authorizing user’s resource access within various components of Hadoop ecosystem. This session will provide a deep dive into Ranger framework and a cook-book for extending Ranger to do authorization / auditing on resource access to external applications, including technical details of Rest APIs, Ranger policy engine and enriching authorization requests, with a demo of a sample application.We will then demonstrate a real-world example of how Ranger has simplified security enforcement for Hadoop-native MPP SQL engine like Apache HAWQ (incubating),which previously used its built-in Postgres-like authorization mechanisms. The integration design includes a Ranger Plugin Service that allows transparent authorization API calls between C-based Apache HAWQ and Java-based Apache Ranger.

Open Policy AgentTorin Sandall

Facebook Messages & HBase强王

The document discusses Facebook's use of HBase to store messaging data. It provides an overview of HBase, including its data model, performance characteristics, and how it was a good fit for Facebook's needs due to its ability to handle large volumes of data, high write throughput, and efficient random access. It also describes some enhancements Facebook made to HBase to improve availability, stability, and performance. Finally, it briefly mentions Facebook's migration of messaging data from MySQL to their HBase implementation.

Improve Monitoring and Observability for Kubernetes with OSS toolsNilesh Gule

Monitoring kubernetes with prometheusBrice Fernandes

Monitoring containerised apps creates a whole new set of challenges that traditional monitoring systems struggle with. In this talk, Brice Fernandes from Weaveworks will introduce and demo the open source Prometheus monitoring toolkit and its integration with Kubernetes. After this talk, you'll be able to use Prometheus to monitor your microservices on a Kubernetes cluster. We'll cover: - An introduction to Kubernetes to manage containers; - The monitoring maturity model; - An overview of whitebox and blackbox monitoring; - Monitoring with Prometheus; - Using PromQL (the Prometheus Query Language) to monitor your app in a dynamic system

Introduction to Open Telemetry as Observability LibraryTonny Adhi Sabastian

- OpenTelemetry is a unified observability library that provides APIs, SDKs, and middleware for generating and exporting traces, metrics, and logs. It combines OpenCensus and OpenTracing into a single open source project. - The OpenTelemetry architecture includes APIs, SDKs, a collector, and exporters. The SDK implements the API to generate traces and metrics. The collector receives telemetry data from the SDK and exports it to backends like Jaeger and Prometheus. - The document demonstrates how to instrument a Go application with OpenTelemetry tracing by creating spans from HTTP requests and exporting them to Jaeger for analysis.

Elastic-EngineeringAraf Karsh Hamid

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

Flink Forward San Francisco 2022. With a real-time processing engine like Flink and a transactional storage layer like Hudi, it has never been easier to build end-to-end low-latency data platforms connecting sources like Kafka to data lake storage. Come learn how to blend Lakehouse architectural patterns with real-time processing pipelines with Flink and Hudi. We will dive deep on how Flink can leverage the newest features of Hudi like multi-modal indexing that dramatically improves query and write performance, data skipping that reduces the query latency by 10x for large datasets, and many more innovations unique to Flink and Hudi. by Ethan Guo & Kyle Weller

ObservabilityDiego Pacheco

This document discusses concepts related to observability including Prometheus, ELK stack, OpenTracing, and Victoria Metrics. It provides examples of setting up Prometheus and Grafana to monitor metrics from applications instrumented with exporters. It also demonstrates setting up Filebeat, Logstash and Elasticsearch (ELK stack) to monitor logs and send them to Elasticsearch. Additionally, it shows how to implement OpenTracing in a Java application and visualize traces using Jaeger. Finally, it outlines an exercise to build a microservices ecommerce application incorporating logging, metrics and tracing using the discussed tools.

Apache kafka 관리와 모니터링JANGWONSEO4

Ozone and HDFS's EvolutionDataWorks Summit

HDFS has several strengths: horizontally scale its IO bandwidth and scale its storage to petabytes of storage. Further, it provides very low latency metadata operations and scales to over 60K concurrent clients. Hadoop 3.0 recently added Erasure Coding. One of HDFS’s limitations is scaling a number of files and blocks in the system. We describe a radical change to Hadoop’s storage infrastructure with the upcoming Ozone technology. It allows Hadoop to scale to tens of billions of files and blocks and, in the future, to every larger number of smaller objects. Ozone fundamentally separates the namespace layer and the block layer allowing new namespace layers to be added in the future. Further, the use of RAFT protocol has allowed the storage layer to be self-consistent. We show how this technology helps a Hadoop user and also what it means for evolving HDFS in the future. We will also cover the technical details of Ozone. Speaker: Sanjay Radia, Chief Architect, Founder, Hortonworks

Using ClickHouse for ExperimentationGleb Kanterov

This document discusses using ClickHouse for experimentation and metrics at Spotify. It describes how Spotify built an experimentation platform using ClickHouse to provide teams interactive queries on granular metrics data with low latency. Key aspects include ingesting data from Google Cloud Storage to ClickHouse daily, defining metrics through a centralized catalog, and visualizing metrics and running queries using Superset connected to ClickHouse. The platform aims to reduce load on notebooks and BigQuery by serving common queries directly from ClickHouse.

Gitlab, GitOps & ArgoCDHaggai Philip Zagury

This document discusses improving the developer experience through GitOps and ArgoCD. It recommends building developer self-service tools for cloud resources and Kubernetes to reduce frustration. Example GitLab CI/CD pipelines are shown that handle releases, deployments to ECR, and patching apps in an ArgoCD repository to sync changes. The goal is to create faster feedback loops through Git operations and automation to motivate developers.

Elastic Observability keynoteElasticsearch

The Apache Spark File Format EcosystemDatabricks

Big data conference europe real-time streaming in any and all clouds, hybri...Timothy Spann

Biography Tim Spann is a Principal DataFlow Field Engineer at Cloudera where he works with Apache NiFi, MiniFi, Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. Talk Real-Time Streaming in Any and All Clouds, Hybrid and Beyond Today, data is being generated from devices and containers living at the edge of networks, clouds and data centers. We need to run business logic, analytics and deep learning at the scale and as events arrive. Tools: Apache Flink, Apache Pulsar, Apache NiFi, MiNiFi, DJL.ai Apache MXNet. References: https://www.datainmotion.dev/2019/11/introducing-mm-flank-apache-flink-stack.html https://www.datainmotion.dev/2019/08/rapid-iot-development-with-cloudera.html https://www.datainmotion.dev/2019/09/powering-edge-ai-for-sensor-reading.html https://www.datainmotion.dev/2019/05/dataworks-summit-dc-2019-report.html https://www.datainmotion.dev/2019/03/using-raspberry-pi-3b-with-apache-nifi.html Source Code: https://github.com/tspannhw/MmFLaNK FLiP Stack StreamNative

Scenic City Summit (2021): Real-Time Streaming in any and all clouds, hybrid...Timothy Spann

More Related Content

What's hot (20)

RedisConf18 - Redis on Google Cloud PlatformRedis Labs

Apache Iceberg: An Architectural Look Under the CoversScyllaDB

KSQL - Stream Processing simplified!Guido Schmutz

Elk Caleb Wang

Elk - An introductionHossein Shemshadi

Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...DataWorks Summit

Open Policy AgentTorin Sandall

Facebook Messages & HBase强王

Improve Monitoring and Observability for Kubernetes with OSS toolsNilesh Gule

Monitoring kubernetes with prometheusBrice Fernandes

Introduction to Open Telemetry as Observability LibraryTonny Adhi Sabastian

Elastic-EngineeringAraf Karsh Hamid

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

ObservabilityDiego Pacheco

Apache kafka 관리와 모니터링JANGWONSEO4

Ozone and HDFS's EvolutionDataWorks Summit

Using ClickHouse for ExperimentationGleb Kanterov

Gitlab, GitOps & ArgoCDHaggai Philip Zagury

Elastic Observability keynoteElasticsearch

The Apache Spark File Format EcosystemDatabricks

RedisConf18 - Redis on Google Cloud PlatformRedis Labs

Apache Iceberg: An Architectural Look Under the CoversScyllaDB

KSQL - Stream Processing simplified!Guido Schmutz

Elk Caleb Wang

Elk - An introductionHossein Shemshadi

Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...DataWorks Summit

Open Policy AgentTorin Sandall

Facebook Messages & HBase强王

Improve Monitoring and Observability for Kubernetes with OSS toolsNilesh Gule

Monitoring kubernetes with prometheusBrice Fernandes

Introduction to Open Telemetry as Observability LibraryTonny Adhi Sabastian

Elastic-EngineeringAraf Karsh Hamid

How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward

ObservabilityDiego Pacheco

Apache kafka 관리와 모니터링JANGWONSEO4

Ozone and HDFS's EvolutionDataWorks Summit

Using ClickHouse for ExperimentationGleb Kanterov

Gitlab, GitOps & ArgoCDHaggai Philip Zagury

Elastic Observability keynoteElasticsearch

The Apache Spark File Format EcosystemDatabricks

Similar to Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020 (20)

Big data conference europe real-time streaming in any and all clouds, hybri...Timothy Spann

Scenic City Summit (2021): Real-Time Streaming in any and all clouds, hybrid...Timothy Spann

Introduction to Apache Mesos and DC/OSSteve Wong

DevOps and Continuous Delivery with CloudFoundryJohannes Rudolph

This document discusses DevOps with Cloud Foundry. It provides an overview of Cloud Foundry, a PaaS that aims to achieve DevOps goals like zero downtime, automated scalability and fast deployments. It demonstrates how Cloud Foundry can be used for continuous delivery by automating deployments to different environments with tools like the CF CLI and Concourse CI. Continuous deployment with Cloud Foundry allows for fast, frequent deployments through features like blue-green deployments and management of services.

Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022HostedbyConfluent

Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022 Modern streaming use cases are generating massive amounts of data - much of it needs to be organized and queried over time. The sheer amount and complexity of this problem presents new challenges for data engineers and developers alike. To solve this problem Apache Kafka and MongoDB Time Series collections are a powerful combination. In this talk, Kenny Gorman and Elena Cuevas will present how Apache Kafka on Confluent Cloud can stream massive amounts of data to Time Series Collections via the MongoDB Connector for Apache Kafka. Elena and Kenny will discuss the required configuration details and critical components of Confluent Cloud and MongoDB Atlas as well as some tips, tricks and best practices. You will leave armed with the knowledge of how Confluent Cloud, Apache Kafka, MongoDB Atlas, and Time Series collections fit into your event-driven architecture.

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...Cisco DevNet

OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs

AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead? At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them. In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.

.NET Cloud-Native Bootcamp- Los AngelesVMware Tanzu

Microsoft Azure Explained - Hitesh D KeshariaHARMAN Services

Microsoft Azure is a cloud computing platform that allows users to build, deploy, and manage applications and services through a global network of Microsoft-managed data centers. It provides infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Azure offers various computing, web & mobile, data & storage, analytics, networking, and other services to help developers build and manage applications in the cloud. It aims to offer reliability, scalability, and easy management to users while ensuring a high availability rate of 99.95%.

Continuous delivery and DevOps with CloudFoundryJohannes Rudolph

Gs08 modernize your data platform with sql technologies wash dcBob Ward

The document discusses the challenges of modern data platforms including disparate systems, multiple tools, high costs, and siloed insights. It introduces the Microsoft Data Platform as a way to manage all data in a scalable and secure way, gain insights across data without movement, utilize existing skills and investments, and provide consistent experiences on-premises, in the cloud, and hybrid environments. Key elements of the Microsoft Data Platform include SQL Server, Azure SQL Database, Azure SQL Data Warehouse, Azure Data Lake, and Analytics Platform System.

Music city data Hail Hydrate! from stream to lakeTimothy Spann

Service fabric and azure service fabric meshMikkel Mørk Hegnhøj

Service Fabric is an open-source distributed systems platform from Microsoft for packaging, deploying and managing distributed applications and services at scale. Azure Service Fabric Mesh is a new fully-managed platform that allows developing and running microservices applications without having to manage infrastructure. Key features of Service Fabric Mesh include serverless infrastructure, lifecycle management, intelligent traffic routing, and health monitoring. It allows building applications using any programming language or framework that can run in containers.

The Never Landing Stream with HTAP and StreamingTimothy Spann

Developing Enterprise Applications for the Cloud, from Monolith to MicroserviceJack-Junjie Cai

Developing Enterprise Applications for the Cloud,from Monolith to MicroservicesDavid Currie

Presented at IBM InterConnect 2105. Is your next enterprise application ready for the cloud? Do you know how to build the kind of low-latency, highly available, highly scalable, omni-channel, micro-service modern-day application that customers expect? This introductory presentation will cover what it takes to build such an application using the multiple language runtimes and composing services offered on IBM Bluemix cloud.

.NET Cloud-Native BootcampVMware Tanzu

Zero to 1000+ Applications - Large Scale CD Adoption at Cisco with Spinnaker ...DevOps.com

As part of its Cloud-native transformation, Cisco needed to modernize its software delivery process. Scalability, multi-cloud deployment to its OpenShift environment and public clouds, and the ability to support Cisco’s extensive policy, compliance, and security requirements made open source Spinnaker a logical choice for a modern continuous delivery platform. As one of the world’s top technology providers with one of the largest and most diverse software development organizations, Cisco had to overcome some unique challenges to be able to onboard 10,000+ developers, 1000+ monolithic and non-cloud native applications, and achieve the high availability and reliability needed to support mission-critical production applications. Join us for this new webinar as Balaji Siva, VP of Products at OpsMx engages Anil Anaberumutt, IT architect at Cisco, and Red Hat Sr. Solutions Architect, Vikas Grover, in a discussion about Cisco’s CD challenges and the lessons learned, best practices implemented, and key results achieved on their CD transformation journey from zero to over 1000 applications.

SqlvikingJonn Callahan

SQLViking is a post exploitation tool written in Python focused on leveraging unencrypted connections between database and web servers. It is comprised of two pieces: one passive and one active. The passive piece, dubbed Scout, sits on the wire and silently collects information passed between database servers and clients. The active piece, Pillage, leverages TCP injection to run arbitrary queries against a database without credentials or man in the middling. SQLViking was designed with extensibility in mind allowing the open source community to easily add support for new databases without needing to touch any of the actual logic of the tool itself via Python's abstract base classes. This talk will cover how the tool works from a functional perspective as well as existing and future features. It will also discuss the root issue which allows this tool to work and how to protect yourself against such an attack.

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...confluent

(Bruno Simic, Solutions Engineer, Couchbase) Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Big data conference europe real-time streaming in any and all clouds, hybri...Timothy Spann

Scenic City Summit (2021): Real-Time Streaming in any and all clouds, hybrid...Timothy Spann

Introduction to Apache Mesos and DC/OSSteve Wong

DevOps and Continuous Delivery with CloudFoundryJohannes Rudolph

Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022HostedbyConfluent

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...Cisco DevNet

OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs

.NET Cloud-Native Bootcamp- Los AngelesVMware Tanzu

Microsoft Azure Explained - Hitesh D KeshariaHARMAN Services

Continuous delivery and DevOps with CloudFoundryJohannes Rudolph

Gs08 modernize your data platform with sql technologies wash dcBob Ward

Music city data Hail Hydrate! from stream to lakeTimothy Spann

Service fabric and azure service fabric meshMikkel Mørk Hegnhøj

The Never Landing Stream with HTAP and StreamingTimothy Spann

Developing Enterprise Applications for the Cloud, from Monolith to MicroserviceJack-Junjie Cai

Developing Enterprise Applications for the Cloud,from Monolith to MicroservicesDavid Currie

.NET Cloud-Native BootcampVMware Tanzu

Zero to 1000+ Applications - Large Scale CD Adoption at Cisco with Spinnaker ...DevOps.com

SqlvikingJonn Callahan

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...confluent

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

Recently uploaded (20)

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

The MCP (Model Context Protocol) is a framework designed to manage context and interaction within complex systems. This SlideShare presentation will provide a detailed overview of the MCP Model, its applications, and how it plays a crucial role in improving communication and decision-making in distributed systems. We will explore the key concepts behind the protocol, including the importance of context, data management, and how this model enhances system adaptability and responsiveness. Ideal for software developers, system architects, and IT professionals, this presentation will offer valuable insights into how the MCP Model can streamline workflows, improve efficiency, and create more intuitive systems for a wide range of use cases.

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Artificial intelligence is changing how businesses operate. Companies are using AI agents to automate tasks, reduce time spent on repetitive work, and focus more on high-value activities. Noah Loul, an AI strategist and entrepreneur, has helped dozens of companies streamline their operations using smart automation. He believes AI agents aren't just tools—they're workers that take on repeatable tasks so your human team can focus on what matters. If you want to reduce time waste and increase output, AI agents are the next move.

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

AI and Data Privacy in 2025: Global TrendsInData Labs

In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy. This infographic contains: -AI and data privacy: Key findings -Statistics on AI data privacy in the today’s world -Tips on how to overcome data privacy challenges -Benefits of AI data security investments. Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.

Procurement Insights Cost To Value Guide.pptxJon Hansen

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

Cyber Awareness overview for 2025 month of securityriccardosl1

How Can I use the AI Hype in my Business Context?Daniel Lehner

𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨? Everyone’s talking about AI but is anyone really using it to create real value? Most companies want to leverage AI. Few know 𝗵𝗼𝘄. ✅ What exactly should you ask to find real AI opportunities? ✅ Which AI techniques actually fit your business? ✅ Is your data even ready for AI? If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Build Your Own Copilot & Agents For DevsBrian McKeiver

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

What is Model Context Protocol(MCP) - The new technology for communication bw...Vishnu Singh Chundawat

Noah Loul Shares 5 Steps to Implement AI Agents for Maximum Business Efficien...Noah Loul

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

AI and Data Privacy in 2025: Global TrendsInData Labs

Procurement Insights Cost To Value Guide.pptxJon Hansen

Semantic Cultivators : The Critical Future Role to Enable AIartmondano

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Cyber Awareness overview for 2025 month of securityriccardosl1

How Can I use the AI Hype in my Business Context?Daniel Lehner

Cybersecurity Identity and Access Solutions using Azure ADVICTOR MAESTRE RAMIREZ

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Build Your Own Copilot & Agents For DevsBrian McKeiver

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

Into The Box Conference Keynote Day 1 (ITB2025)Ortus Solutions, Corp

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

How analogue intelligence complements AIPaul Rowe

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020

1. Feed Your SIEM Smart with Kafka Connect Vitalii Rudenskyi Security Architect, McKesson Corp

2. Motivation and Background

3. Why We Started It’s not a mistake to make a mistake, but it’s a mistake to repeat the same mistake

4. Requirements ★ Logs collection to be vendor agnostic ★ Feed different analytics tools ★ Data filtering and cleaning ★ Existing data retention requirements ★ Scalable and highly available

5. Anticipated Challenges ★ Amount of data and number of sources ★ Variety of different formats ★ SaaS/PaaS applications and public Clouds ★ Cloud SIEM

6. Kafka Connect

7. The Three Keys That Open The Door

8. How to Collect Data

9. The Three Keys ‘Push’ Connector ‘Pull’ Connector Transformations Library

10. Push - NettySource Connector ★ Unified: “all-in-one” different transport and protocols ★ Scalable: multiple tasks supported ★ Available: works behind a LB (health checks supported) ★ Configurable: multiple protocols (plain text, syslog, http, snmp, netflow) ★ Customizable: custom implementations supported

11. NettySource Connector Common deployment model

12. Pull - PollableAPIClient Connector ★ Simple: easy new connectors development ★ Scalable: multiple tasks/partitions supported ★ Conﬁgurable: interval and scheduled (cron-style) polls, retry/backoff, resettable offsets ★ Customizable: custom implementations supported

13. PollableAPIClient Connector

14. PollableAPIClient Connector 25+ ApiClient implementations

15. Transformations Library ★ Can transform different parts of a kafka message ★ Supports “if” conditions

16. Takeaways

17. Highly Available NettySource Connector

18. Headers All The Way ★ Track origin of the data ★ Conditional routing ★ Tagging in ‘chained’ transformations ★ SIEM specific metadata

19. Syslog: Take Port 514 Under Control Multi-rules RegexRouter config

20. Data archiving solution ★ Pair of source and sink connectors ★ One connector for all topics ★ Compressed ★ Easy to restore

21. ★ Monitoring is essential ★ Keep original data ★ Be cautious when using “heavy” transformations in source connectors

22. 5 kafka clusters 20+ kafka connect clusters 530+ deployed connectors 7+TB of data daily

23. What is the next...

24. We share! ★ NettySource Connector https://github.com/vrudenskyi/kafka-connect-netty-source ★ PollableAPIClient Connector https://github.com/vrudenskyi/kafka-connect-pollable-source https://github.com/vrudenskyi/kafka-connect-api-clients ★ Transformations Library https://github.com/vrudenskyi/kafka-connect-transform

25. Q & [email protected] https://www.linkedin.com/in/vrudenskyi/

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020

Recommended

More Related Content

What's hot (20)

Similar to Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020 (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020