DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020

Sep 10, 20200 likes1,348 views

DataOps challenges us to build data experiences in a repeatable way. For those with Kafka, this means finding a means of deploying flows in an automated and consistent fashion. The challenge is to make the deployment of Kafka flows consistent across different technologies and systems: the topics, the schemas, the monitoring rules, the credentials, the connectors, the stream processing apps. And ideally not coupled to a particular infrastructure stack. In this talk we will discuss the different approaches and benefits/disadvantages to automating the deployment of Kafka flows including Git operators and Kubernetes operators. We will walk through and demo deploying a flow on AWS EKS with MSK and Kafka Connect using GitOps practices: including a stream processing application, S3 connector with credentials held in AWS Secrets Manager.

for a Kafka streaming platform
DataOps automation

Presenters
Andrew Stevenson
CTO Lenses.io
Spiros Economakis
DevOps Cloud Lead Lenses.io

Delivering data intensity on Kafka
Lenses.io

What is DataOps & Lenses.io
How to automate your data landscape
GitOps demo
Agenda

Deploying Kafka isn’t interesting any more?

Tech Intensity
Use commodity infrastructure
Don’t reinvent
Focus on building data landscape

If not kafka, what are we automating?
The important bit...
Data Intensity
Tech Intensity

Pushing
Are sure you want to give access?
Who resolves the desired state?
Who applies the desired state?

GitOps Benefits
Focus build reusable apps - config driven - less ops
Reduced attack surface to sensitive environments
Immutable
Auditable
Rollback
Separation of concerns
Use any Git server - Don’t need k8s
Operator enforces multi-tenancy and governance

GitOps for Kafka
Infrastructure deployment is now a commodity - Terraform, Cloudformation, etc.
Tech intensity
Repeatable, rapid deployment of the data landscape
Topics
Schemas
ACLs
Alert rules
Data policies
Applications
Data intensity

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020

LEARN MORE
All-in-one Lenses.io DataOps
workspace & Kafka docker
container
Lenses.io workspace with
Kafka in the cloud
lenses.io/start
launchpass.com/lensesio
https://launchpass.com/lensesio

for a Kafka streaming platform
DataOps automation
Tuesday, August 25 2020
07:30 PDT / 14h30 GMT
events.kafka-summit.org/2020-schedule

The document provides an overview of leveraging mainframe data for modern analytics using Attunity Replicate and Confluent streaming platform powered by Apache Kafka. It discusses the history of mainframes and data migration, how Attunity enables real-time data migration from mainframes, the Confluent streaming platform for building applications using data streams, and how Attunity and Confluent can be combined to modernize analytics using mainframe data streams. Use cases discussed include query offloading and cross-system customer data integration.

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Data in Motion: Building Stream-Based Architectures with Qlik Replicate & Kaf...HostedbyConfluent

The challenge with today’s “data explosion” is finding the most appropriate answer to the question, “So where do I put my data?” while avoiding the longer-term problem: data warehouses, data lakes, cloud storage, NoSQL databases, … are often the places where “big” data goes to die. Enter Physics 101, and my corollary to Newton’s First Law of Motion: Data in motion tends to stay in motion until it comes rest on disk. Similarly, if data is at rest, it will remain at rest until an external “force” puts it in motion again. Data inevitably comes to rest at some point. Without “external forces”, data often gets lost or becomes stale where it lands. “Modern” architectures tend to involve data pipelines where downstream consumers of data make use of data generated upstream, often with built-for-purpose repositories at each stage. This session will explore how data that has come to rest can be put in motion again; how Kafka can keep it in motion longer; and how pipelined architectures might be created to make use of that data.

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...HostedbyConfluent

Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...HostedbyConfluent

Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...HostedbyConfluent

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

Maximize the Business Value of Machine Learning and Data Science with Kafka (...confluent

Today, many companies that have lots of data are still struggling to derive value from machine learning (ML) and data science investments. Why? Accessing the data may be difficult. Or maybe it’s poorly labeled. Or vital context is missing. Or there are questions around data integrity. Or standing up an ML service can be cumbersome and complex. At Nuuly, we offer an innovative clothing rental subscription model and are continually evolving our ML solutions to gain insight into the behaviors of our unique customer base as well as provide personalized services. In this session, I’ll share how we used event streaming with Apache Kafka® and Confluent Cloud to address many of the challenges that may be keeping your organization from maximizing the business value of machine learning and data science. First, you’ll see how we ensure that every customer interaction and its business context is collected. Next, I’ll explain how we can replay entire interaction histories using Kafka as a transport layer as well as a persistence layer and a business application processing layer. Order management, inventory management, logistics, subscription management – all of it integrates with Kafka as the common backbone. These data streams enable Nuuly to rapidly prototype and deploy dynamic ML models to support various domains, including pricing, recommendations, product similarity, and warehouse optimization. Join us and learn how Kafka can help improve machine learning and data science initiatives that may not be delivered to their full potential.

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...HostedbyConfluent

SIEM platforms are essential to the new cybersecurity paradigm and data collection layer is a very important piece of it. When you deliver a new platform, you can easily get lost in a variety of different vendors and solutions, too many challenges are facing. What if I change vendors, will I keep my data? How to feed multiple tools with the same data? How to collect data from custom apps and services? How to pay less for an expensive platform? How to keep data without a huge cost? Join us if you are looking for the answers. In this session, you will learn how we replaced the vendor-provided data collection layer with kafka connect and the lessons we learnt. After the talk you will know: - architecture and real-life examples of the flexible and highly available data collection platform - custom connectors that do most of the work for us and how to extend the connectors to consume new data, we made them open sourced - easy way to receive data from thousands of servers and many cloud services - how to archive data at low cost You will leave armed with a set of free tools and recipes to build a truly vendor-agnostic data collection platform. It will allow you to take you SIEM costs under control. You will feed your analytics tools with what they need and archive the rest at low cost. You will feed your SIEM smart!

How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020HostedbyConfluent

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...HostedbyConfluent

The document discusses Kafka in the context of cloud platforms and open source communities. It describes several Apache projects that can be used with Kafka, such as Apache NiFi for data collection, Apache Flink for stream processing, and Apache Ranger for security. It also outlines features of Cloudera's platform for managing Kafka deployments, including unified governance tools, monitoring, and services to simplify operations. Finally, it discusses how Kafka can be deployed across cloud, on-premise, and hybrid environments with auto-scaling and other management capabilities.

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...confluent

This document discusses challenges with keeping a metadata repository current using event-driven updates from data sources. It describes how using Apache Kafka and the Debezium connector to capture changes from database "outbox" tables that mirror system catalog metadata tables allows pushing metadata deltas to the repository in real-time. This overcomes limitations of log-based and query-based CDC approaches when applied directly to database system tables.

Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent

Due to explosion of IoT, we have streaming data that needs to be processed in real-time. This needs to be made available for applications as well as analytics scenarios such as anomaly detection. This workshop presents a solution using Confluent Cloud on Azure, Azure Cosmos DB and Azure Synapse Analytics which can be connected in a secure way within Azure VNET using Azure Private link configured on Kafka clusters.

Real-Time Analytics with Confluent and MemSQLSingleStore

This document discusses enabling real-time analytics for IoT applications. It describes how industries like auto, transportation, energy, warehousing and logistics, and healthcare need real-time analytics to handle streaming data from IoT sensors. It also discusses how Confluent's Kafka stream processing platform can be used to build applications that ingest IoT data at high speeds, transform the data, and power real-time analytics and user interfaces. MemSQL's in-memory database is presented as a fast and scalable storage option to support real-time analytics on the large volumes of IoT data.

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...HostedbyConfluent

Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better. The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka. We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices. Some potential takeaways for the discerning audience: 1. Opportunities in a platform approach to Event Driven Architecture in the enterprise 2. Adopting a product mindset around Data & Event Streams 3. Seeking harmony with allied enterprise applications

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

PCAP Graphs for Cybersecurity and System TuningDr. Mirko Kämpf

This document discusses analyzing network traffic patterns in Hadoop clusters. Packet capture data was collected from example Hadoop workloads and analyzed using Gephi. Initial results show the network structure and communication between nodes for batch processing (TeraSort) and real-time streaming (Twitter collection). Further analysis aims to classify components, understand dependencies, and identify anomalies over time to better understand typical and atypical workload behavior.

user Behavior Analysis with Session Windows and Apache Kafka's Streams APIconfluent

For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior. While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement. In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent

This document discusses streaming data between Confluent Cloud and MongoDB Atlas. It provides an overview of MongoDB Atlas and its fully managed database capabilities in the cloud. It then demonstrates how to stream data from a Python generator application to MongoDB Atlas using Confluent Cloud and its connectors. The presentation concludes by providing a reference architecture for connecting Confluent Platform to MongoDB.

Data Integration with Apache Kafka: What, Why, HowPat Patterson

Presented at Orange County Advanced Analytics and Big Data Meetup, June 21 2019. Apache Kafka has fast become the dominant messaging technology for the enterprise; if you're a data scientist or data engineer and you have not yet worked with Kafka, that situation will likely change soon! In this session, Pat Patterson, director of evangelism at StreamSets, explains what Kafka is, why it has disrupted the previous generation of messaging products, and how you can use open source products to build dataflow pipelines with Kafka, without writing code.

INTRODUCING: CREATE PIPELINESingleStore

This document introduces MemSQL Pipelines, which allow for exactly-once data ingestion semantics when streaming data from Kafka into MemSQL. MemSQL Pipelines provide a native way to extract, transform, and load external data into MemSQL tables. They offer a scalable and highly performant ETL process across a distributed cluster. The document explains streaming semantics like at least once and exactly once delivery, and how MemSQL Pipelines coordinate with Kafka to enable exactly-once ingestion through offset tracking. It presents the architecture of MemSQL Pipelines and demonstrates their use through a live demo.

Enterprise Metadata IntegrationDr. Mirko Kämpf

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent

Apache Hudi is a data lake platform, that provides streaming primitives (upserts/deletes/change streams) on top of data lake storage. Hudi powers very large data lakes at Uber, Robinhood and other companies, while being pre-installed on four major cloud platforms. Hudi supports exactly-once, near real-time data ingestion from Apache Kafka to cloud storage, which is typically used in-place of a S3/HDFS sink connector to gain transactions and mutability. While this approach is scalable and battle-tested, it can only ingest data in mini batches, leading to lower data freshness. In this talk, we introduce a Kafka Connect Sink Connector for Apache Hudi, which writes data straight into Hudi's log format, making the data immediately queryable, while Hudi's table services like indexing, compaction, clustering work behind the scenes, to further re-organize for better query performance.

Disrupting Big Data with Apache Spark in the CloudJen Aman

This document discusses the challenges of big data analytics and how Apache Spark and Databricks can help address them. It summarizes that: 1) There is a gap between the growth of data and ability to perform real-time analytics on that data due to challenges in managing infrastructure, empowering teams, and establishing production-ready applications. 2) Databricks provides a cloud-hosted platform that uses Apache Spark to allow for just-in-time processing of data across storage silos, with an integrated workspace for interactive exploration, machine learning, and production-ready workflows. 3) Databricks Enterprise Security provides an end-to-end security solution for Apache Spark to address challenges in securing file

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Implyconfluent

Presenters: Rachel Pedreschi, Senior Director, Solutions Engineering, Imply.io + Josh Treichel, Partner Solutions Architect, Confluent Analytic pipelines running purely on batch processing systems can suffer from hours of data lag, resulting in accuracy issues with analysis and overall decision-making. Join us for a demo to learn how easy it is to integrate your Apache Kafka® streams in Apache Druid (incubating) to provide real-time insights into the data. In this online talk, you’ll hear about ingesting your Kafka streams into Imply’s scalable analytic engine and gaining real-time insights via a modern user interface. Register now to learn about: -The benefits of combining a real-time streaming platform with a comprehensive analytics stack -Building an analytics pipeline by integrating Confluent Platform and Imply -How KSQL, streaming SQL for Kafka, can easily transform and filter streams of data in real time -Querying and visualizing streaming data in Imply -Practical ways to implement Confluent Platform and Imply to address common use cases such as analyzing network flows, collecting and monitoring IoT data and visualizing clickstream data Confluent Platform, developed by the creators of Kafka, enables the ingest and processing of massive amounts of real-time event data. Imply, the complete analytics stack built on Druid, can ingest, store, query and visualize streaming data from Confluent Platform, enabling end-to-end real-time analytics. Together, Confluent and Imply can provide low latency data delivery, data transform, and data querying capabilities to power a range of use cases.

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...Pat Patterson

On a typical day we see hundreds of downloads of StreamSets Data Collector, our open source data integration tool. We used to wrangle our download logs using a combination of the AWS S3 command line, sed, grep, awk and other tools, all run from a shell script (on my laptop!) once a week. This was a classic example of a brittle, hard to maintain, custom data integration. One day it dawned on me, "This is crazy, we have a tool that can do all this!". In this session, I'll explain how I built a dataflow pipeline to stream content delivery network (CDN) logs from S3 to MySQL in real-time, allowing us to gain valuable insights into our open source community. You'll also learn how we use the same techniques to not only gain insights into our community on Slack, but also build tools to better serve them.

Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...confluent

Responding to a global pandemic presents a unique set of technical and public health challenges. The real challenge is the ability to gather data coming in via many data streams in variety of formats influences the real-world outcome and impacts everyone. The Centers for Disease Control and Prevention CELR (COVID Electronic Lab Reporting) program was established to rapidly aggregate, validate, transform, and distribute laboratory testing data submitted by public health departments and other partners. Confluent Kafka with KStreams and Connect play a critical role in program objectives to: o Track the threat of COVID-19 virus o Provide comprehensive data for local, state, and federal response o Better understand locations with an increase in incidence

Operational Analytics on Event Streams in Kafkaconfluent

Speaker: Anirudh Ramanthan, Product Manager, Rockset Tracking key events and analyzing these event streams are critical to many enterprises. We highlight how organizations are using Apache Kafka® as a fast, reliable event streaming platform alongside Rockset, a serverless search and analytics engine, to create stateful microservices to analyze their event streams. In this talk, we will discuss a stateful microservices architecture, where events from multiple channels are collected and streamed into Kafka and continuously ingested into Rockset with no explicit schema or metadata specification required. Developers then use serverless compute frameworks, like AWS Lambda, in conjunction with serverless data management from Rockset to build microservices to derive insights on the data from Kafka. Organizations can leverage this pattern to support low-latency queries on event streams, providing immediate insight on their business.

All the Ops: DataOps with GitOps for Streaming data on Kafka and KubernetesDevOps.com

Running Apache Kafka and Kubernetes is synonymous with containerized real time data. Many users have adopted the pairing to deploy and manage individual distributed real time applications. While Kubernetes allows developers to scale applications in microservices quicker, there are still productivity blockers such as visibility and governance. Enter DataOps. In this webinar, you'll learn how to: Enhance the productivity of your Kafka & Kubernetes stream with DataOps Enable enterprise adoption and scaling Govern & secure your stream

[Capitole du Libre] #serverless -  mettez-le en oeuvre dans votre entreprise...Ludovic Piot

Tout comme le Cloud IaaS avant lui, le serverless promet de faciliter le succès de vos projets en accélérant le Time to Market et en fluidifiant les relations entre Devs et Ops. Mais sa mise en œuvre au sein d’une entreprise reste complexe et coûteuse. Après 2 ans à mettre en place des plateformes managées de ce type, nous partagons nos expériences de ce qu’il faut faire pour mettre en œuvre du serverless en entreprise, en évitant les douleurs et en limitant les contraintes au maximum. Tout d’abord l’architecture technique, avec 2 implémentations très différentes : Kubernetes et Helm d’un côté, Clever Cloud on-premise de l’autre. Ensuite, la mise en place et l’utilisation d’OpenFaaS. Comment tester et versionner du Function as a Service. Mais aussi les problématiques de blue/green deployment, de rolling update, d’A/B testing. Comment diagnostiquer rapidement les dépendances et les communications entre services. Enfin, en abordant les sujets chers à la production : * vulnerability management et patch management, * hétérogénéïté du parc, * monitoring et alerting, * gestion des stacks obsolètes, etc.

More Related Content

What's hot (20)

How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020HostedbyConfluent

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...HostedbyConfluent

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...confluent

Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent

Real-Time Analytics with Confluent and MemSQLSingleStore

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...HostedbyConfluent

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

PCAP Graphs for Cybersecurity and System TuningDr. Mirko Kämpf

user Behavior Analysis with Session Windows and Apache Kafka's Streams APIconfluent

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent

Data Integration with Apache Kafka: What, Why, HowPat Patterson

INTRODUCING: CREATE PIPELINESingleStore

Enterprise Metadata IntegrationDr. Mirko Kämpf

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent

Disrupting Big Data with Apache Spark in the CloudJen Aman

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Implyconfluent

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...Pat Patterson

Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...confluent

Operational Analytics on Event Streams in Kafkaconfluent

How a distributed graph analytics platform uses Apache Kafka for data ingesti...HostedbyConfluent

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020HostedbyConfluent

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...HostedbyConfluent

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...confluent

Extracting Value from IOT using Azure Cosmos DB, Azure Synapse Analytics and ...HostedbyConfluent

Real-Time Analytics with Confluent and MemSQLSingleStore

Event & Data Mesh as a Service: Industrializing Microservices in the Enterpri...HostedbyConfluent

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

PCAP Graphs for Cybersecurity and System TuningDr. Mirko Kämpf

user Behavior Analysis with Session Windows and Apache Kafka's Streams APIconfluent

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent

Data Integration with Apache Kafka: What, Why, HowPat Patterson

INTRODUCING: CREATE PIPELINESingleStore

Enterprise Metadata IntegrationDr. Mirko Kämpf

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent

Disrupting Big Data with Apache Spark in the CloudJen Aman

Achieve Sub-Second Analytics on Apache Kafka with Confluent and Implyconfluent

Project Ouroboros: Using StreamSets Data Collector to Help Manage the StreamS...Pat Patterson

Flattening the Curve with Kafka (Rishi Tarar, Northrop Grumman Corp.) Kafka S...confluent

Operational Analytics on Event Streams in Kafkaconfluent

Similar to DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020 (20)

All the Ops: DataOps with GitOps for Streaming data on Kafka and KubernetesDevOps.com

[Capitole du Libre] #serverless -  mettez-le en oeuvre dans votre entreprise...Ludovic Piot

Building Data Intensity with AWS MSK & Lenses.ioLenses.io

This document discusses building data intensity through a DataOps approach. It emphasizes focusing on what matters through streaming data flows, self-service data visibility and governance, and skills. Data platforms need to provide visibility, access to data, integration capabilities, transformation capabilities, and deployment/scaling while being API-driven and allowing for governance, security and multi-tenancy. Technology should enable focus on data intensity. The remainder discusses Lenses.io's product for real-time applications and data operations and its use with AWS MSK for a customer, Vortexa, which tracks oil and gas flows.

OpenStack Preso: DevOps on Hybrid Infrastructurerhirschfeld

2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructuredevopsdaysaustin

The document discusses the need for hybrid infrastructure and hybrid DevOps to manage different cloud platforms and physical infrastructure in a consistent way. It notes that while no single API or platform can meet all needs, AWS dominance means its operational patterns have become the benchmark. The key is developing composable infrastructure modules that can be orchestrated together to provide portability across environments using a common operational process.

Gluecon Preso: Hybrid Container Infrastructurerhirschfeld

Continuous Lifecycle London 2018 Event KeynoteWeaveworks

Today it’s all about delivering velocity without compromising on quality, yet it’s becoming increasingly difficult for organisations to keep up with the challenges of current release management and traditional operations. The demand for developers to own the end-to-end delivery, including operational ownership, is increasing. A “you build it, you own it” development process requires tools that developers know and understand. So I’d like to introduce “GitOps”- an agile software lifecycle for modern applications. In this session, I will discuss these industry challenges, including current CICD trends and how they’re converging with operations and monitoring. I’ll also illustrate the GitOps model, identify best practices and tools to use, and explain how you can benefit from adopting this methodology inherited from best practices going back 10-15 years.

Wasp2 - IoT and Streaming PlatformPaolo Platter

Yannis Zarkadas. Enterprise data science workflows on kubeflowMarynaHoldaieva

This document discusses using GitOps and multi-tenancy to provide an enterprise data science experience on Kubeflow. It describes how to deploy and manage Kubeflow using GitOps to simplify operations and accelerate time to production. GitOps allows storing all configuration in Git for versioning, rollbacks, and collaboration. Kustomize is used to manage configurations and integrate changes from upstream Kubeflow repositories while allowing customizations. This provides reproducibility, isolation of tenants, and easy upgrades in a production-ready Kubeflow cluster.

Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...Lviv Startup Club

Enterprise Application MigrationVMware Tanzu

Episode 4: Operating Kubernetes at Scale with DC/OSMesosphere Inc.

You’ve installed your Kubernetes cluster on DC/OS — now what? Operating Kubernetes efficiently can be challenging. In the final episode of our Kubernetes series, we will share best practices for operating your DC/OS Kubernetes cluster and maintaining performance. During this presentation, Joerg Schad and Chris Gaun show you how to successfully operate Kubernetes at scale in your environment. During this session, we discuss: 1. How to upgrade DC/OS and Kubernetes with no downtime 2. How DC/OS guards against failure and enables fault domains that are resistant to outages within racks, availability zones, or cloud environments 3. How the monitoring and metrics capabilities on DC/OS improve operational analytics and help you get the most from your cluster 4. How cloud bursting extends your on-prem environment with resources from the cloud to handle spikes in your workload

Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersenconfluent

James Watters Kafka Summit NYC 2019 KeynoteJames Watters

The document discusses how Spring Boot and Kafka can form the basis of a new enterprise application platform that enables continuous delivery and efficient scaling through microservices and event-driven architecture. It provides examples of companies like Netflix and T-Mobile that have successfully adopted this approach. The document advocates an "event-first" design and argues this platform approach allows for arbitrary scaling, multi-cloud deployment, and increased developer autonomy and agility.

Containers and KubernetesAltoros

The combination of StackPointCloud with NetApp creates NetApp Kubernetes Service, the industry’s first complete Kubernetes platform for multi-cloud deployments and a complete cloud-based stack for Azure, Google Cloud, AWS, and NetApp HCI. Further, Trident is a fully supported open source project maintained by NetApp, designed from the ground up to help meet the sophisticated persistence demands of containerized applications.

Netflix Cloud Architecture and Open Sourceaspyker

The Never Landing Stream with HTAP and StreamingTimothy Spann

ATO 2018 - What is Serverless Useful For?BerndtJung

This document discusses serverless computing and introduces Dispatch, an open source serverless platform from VMware. It describes how serverless uses functions to provide a serverless computing experience across multiple clouds. Dispatch provides a production-ready functions stack that supports cloud events and can integrate with various identity providers and services. The document demonstrates Dispatch's capabilities through an example using Dispatch and Slack for chat-based automation and management of virtual machines and events. It outlines Dispatch's roadmap, including both standalone and Knative-based versions, and invites the reader to get involved.

End to end Machine Learning using Kubeflow - Build, Train, Deploy and ManageAnimesh Singh

This document discusses Kubeflow, an end-to-end machine learning platform for Kubernetes. It covers various Kubeflow components like Jupyter notebooks, distributed training operators, hyperparameter tuning with Katib, model serving with KFServing, and orchestrating the full ML lifecycle with Kubeflow Pipelines. It also talks about IBM's contributions to Kubeflow and shows how Watson AI Pipelines can productize Kubeflow Pipelines using Tekton.

How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...Lightbend

All the Ops: DataOps with GitOps for Streaming data on Kafka and KubernetesDevOps.com

[Capitole du Libre] #serverless -  mettez-le en oeuvre dans votre entreprise...Ludovic Piot

Building Data Intensity with AWS MSK & Lenses.ioLenses.io

OpenStack Preso: DevOps on Hybrid Infrastructurerhirschfeld

2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructuredevopsdaysaustin

Gluecon Preso: Hybrid Container Infrastructurerhirschfeld

Continuous Lifecycle London 2018 Event KeynoteWeaveworks

Wasp2 - IoT and Streaming PlatformPaolo Platter

Yannis Zarkadas. Enterprise data science workflows on kubeflowMarynaHoldaieva

Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...Lviv Startup Club

Enterprise Application MigrationVMware Tanzu

Episode 4: Operating Kubernetes at Scale with DC/OSMesosphere Inc.

Best Practices for Building Hybrid-Cloud Architectures | Hans Jespersenconfluent

James Watters Kafka Summit NYC 2019 KeynoteJames Watters

Containers and KubernetesAltoros

Netflix Cloud Architecture and Open Sourceaspyker

The Never Landing Stream with HTAP and StreamingTimothy Spann

ATO 2018 - What is Serverless Useful For?BerndtJung

End to end Machine Learning using Kubeflow - Build, Train, Deploy and ManageAnimesh Singh

How to build streaming data pipelines with Akka Streams, Flink, and Spark usi...Lightbend

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

Recently uploaded (20)

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

Build Your Own Copilot & Agents For DevsBrian McKeiver

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts! 📕 Agenda Welcome & Introductions Orchestrator API Overview Exploring the Swagger Interface Test Manager API Highlights Streamlining Automation & Testing with APIs (Demo) Q&A and Open Discussion Perfect for developers, testers, and automation enthusiasts! 👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/ This session streamed live on April 29, 2025, 18:00 CET. Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

This is a Quick Research Guide (QRG). QRGs include the following: - A brief, high-level overview of the QRG topic. - A milestone timeline for the QRG topic. - Links to various free online resource materials to provide a deeper dive into the QRG topic. - Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic. QRGs planned for the series: - Artificial Intelligence QRG - Quantum Computing QRG - Big Data Analytics QRG - Spacecraft Guidance, Navigation & Control QRG (coming 2026) - UK Home Computing & The Birth of ARM QRG (coming 2027) Any questions or comments? - Please contact Arthur Morgan at [email protected]. 100% human made.

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/ HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client. Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience. In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including - Accessing the console - Locating and interpreting log files - Accessing the data folder within the browser’s cache (using OPFS) - Understand the difference between single- and multi-user scenarios - Utilizing Client Clocking

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

I started my online journey with several hosting services before stumbling upon Ai EngineHost. At first, the idea of paying one fee and getting lifetime access seemed too good to pass up. The platform is built on reliable US-based servers, ensuring your projects run at high speeds and remain safe. Let me take you step by step through its benefits and features as I explain why this hosting solution is a perfect fit for digital entrepreneurs.

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots. 📕 Here's what you can expect: - Modeling: Build end-to-end processes using BPMN. - Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes. - Operating: Control process instances with rewind, replay, pause, and stop functions. - Monitoring: Use dashboards and embedded analytics for real-time insights into process instances. This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes. 👨‍🏫 Speaker: Andrei Vintila, Principal Product Manager @UiPath This session streamed live on April 29, 2025, 16:00 CET. Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

How analogue intelligence complements AIPaul Rowe

Build Your Own Copilot & Agents For DevsBrian McKeiver

UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity

IEDM 2024 Tutorial2_Advances in CMOS Technologies and Future Directions for C...organizerofv

Greenhouse_Monitoring_Presentation.pptx.hpbmnnxrvb

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Generative Artificial Intelligence (GenAI) in BusinessDr. Tathagat Varma

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan

HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company

2025-05-Q4-2024-Investor-Presentation.pptxSamuele Fogagnolo

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...SOFTTECHHUB

Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity

Andrew Marnell: Transforming Business Strategy Through Data-Driven InsightsAndrew Marnell

Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020

1. for a Kafka streaming platform DataOps automation

2. Presenters Andrew Stevenson CTO Lenses.io Spiros Economakis DevOps Cloud Lead Lenses.io

3. Delivering data intensity on Kafka Lenses.io

4. What is DataOps & Lenses.io How to automate your data landscape GitOps demo Agenda

5. What do we do?

6. Deploying Kafka isn’t interesting any more?

7. Well trodden path

8. Tech Intensity Use commodity infrastructure Don’t reinvent Focus on building data landscape

9. If not kafka, what are we automating? The important bit... Data Intensity Tech Intensity

10. How are you automating?

11. Pushing Are sure you want to give access? Who resolves the desired state? Who applies the desired state?

12. K8s operator for Kafka Better but...

13. Observe git apply desired state GitOps

14. GitOps for DataOps

15. GitOps Benefits Focus build reusable apps - config driven - less ops Reduced attack surface to sensitive environments Immutable Auditable Rollback Separation of concerns Use any Git server - Don’t need k8s Operator enforces multi-tenancy and governance

16. GitOps for Kafka Infrastructure deployment is now a commodity - Terraform, Cloudformation, etc. Tech intensity Repeatable, rapid deployment of the data landscape Topics Schemas ACLs Alert rules Data policies Applications Data intensity

17. GitOps for Kafka applications Demo

18. Demo architecture

20. Building data intensity DataOps

21. LEARN MORE All-in-one Lenses.io DataOps workspace & Kafka docker container Lenses.io workspace with Kafka in the cloud lenses.io/start launchpass.com/lensesio https://launchpass.com/lensesio

22. for a Kafka streaming platform DataOps automation Tuesday, August 25 2020 07:30 PDT / 14h30 GMT events.kafka-summit.org/2020-schedule

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020

Recommended

More Related Content

What's hot (20)

Similar to DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020 (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros Economakis, Lenses) Kafka Summit 2020