Leveraging Microservices and Apache Kafka to Scale Developer Productivity

Jan 20, 20210 likes400 views

Presenter: Mohinish Shaikh, Software Engineer, Confluent Streaming Audio Podcast: Scaling Developer Productivity with Apache Kafka https://cnfl.io/scaling-developer-productivity-with-apache-kafka

Leveraging
Microservices &
Apache Kafka® to Scale
Developer Productivity
Mohinish Shaikh
@mohinishbasha
https://www.linkedin.com/in/mohinishbasha/

How ...
What ...
Why ...
of Developer
productivity

development ...
process ...
operations ...
engineering ...
devops ...
sre ...
pipelines ...
engineering services ...
platform infrastructure ...
whoami

Developer productivity platform
Quality & Test
Automation
Packaging
Commits
Monitoring, Logging,
Tracing
Infrastructure
Management &
Automation
CI/ CD across Stage, QA,
Integ, Perf, Production
* Productivity
Intelligence
Builds, Releases &
Deployments
Observability
Security, Compliance &
Audit
Developer Productivity
Platform

. . . anything &
everything between
commit to production
and beyond ...

common
tools …
systems ..
frameworks ...
infrastructure …
standards …
architectural patterns ...
automation …
workflows …
workloads ...

Support
polyglot applications ...
quality ...
reliability ...
efficiency ...
environments ...
speed ...
time to market ...
roi ...

products expansion ...
lob ...
complementary services ...
bigger markets ...
partners ...
community ...
compliance ...
laws ...
geography ...
languages ...
culture ...

if ...
sales ...
promises ...
stuff ...
to ...
customers …
Developer Productivity ...
helps the ...
organization ...
Deliver with speed

nasa helios 2 set a record
speed of 157078 mph in 1976

Leveraging Microservices and Apache Kafka to Scale Developer Productivity

Cohesive ...
Hyperconverged ...
self serviced ...
self aware ...
Agnostic ...
Pluggable ...

as organizations grow over time,
things become
monolithic in nature & silo’d

building a productivity
platform that scales while
avoiding monolithic traps &
technical debt is the
challenge

pipelines
containerize tooling ..
source code analysis ..
unit, functional, perf tests ..
builds, releases, packaging ..
deployments, infrastructure ..
automated ..
repeatable ..
de-coupled ..
systems ..
services ..

Learn, adapter, evolve
using
a productivity
intelligence system.

Mgmt. Infrastructure
Productivity platform
DB2
DB1
KV
DWH
Builds
Provisionin
g
KV2
KV3
App2-v2
App5 App7
App1-v2
Application Cluster
App8
DWH
Tests
Kafka
Kafka
Deploymen
ts
DB3
SCM
Productivity Intelligence

Before: Many Ad Hoc Pipelines
Search Security
General Application Architecture
Service Discovery/ KV Logging, Tracing Operational Metrics
Hadoop
Product1: App Data
Warehouse
Provisioning Builds Deployments
Databases
Storage
Interfaces
Product2: App
Databases
Storage
Interfaces
Artifacts
Governance
Test Automation
IaaS
Compliance
Security

Kafka Streams
Kafka Streams
After: Productivity Platform with Kafka
✓ cohesive ✓ agnostic ✓ Self serviceable
Search Security
General Application Architecture
Service Discovery/ KV
Monitoring, Logging,
Tracing
Kafka
Provisioning Builds Deployments
Hadoop Product1: App Product2: App
Data
Warehouse
pipelines running productivity micro services
✓ hyper-converged
Governance
Artifacts
Test automation
Immutable IaaS
Security
Compliance

Management Cluster: future
build container
Tools Service pod
Native Kafka Java
Applications
build container
Management Pipelines Management Cluster
Provisioning
Artifacts
Test Infrastructure
Builds, Release &
Packaging
Deployments
Tools container
Tools Service pod
Native Kafka Java
Applications
Tools container
Productivity Intelligence

Code philosophy – Plan, Apply, Destroy
State: Maintain state of automation & infrastructure at all times.
1 2
Plan
generate a plan of
execution
Execute
Apply changes based
on generated plan.
3
Destroy
At any given point in
time tools should be
able to roll back
KV
Configuration
Keep configuration out
of tooling.

Q & A.
Mohinish Shaikh
@mohinishbasha
https://www.linkedin.com/in/mohinishbasha/

In this all too fabulous talk we will be addressing the wonderful and new wonders of KSQL vs. KStreams. If you are new-ish to Kafka…you may ask yourself, “What is a large Kafka deployment?” And you may tell yourself, “This is not my beautiful KSQL use case!” And you may tell yourself, “This is not my beautiful KStreams use case!” And you may ask yourself, “What is a beautiful Kafka use case?” And you may ask yourself, “Where does that stream process go to?” And you may ask yourself, “Am I right about this architecture? Am I wrong?” And you may say yourself, “My God! What have I done?” In this talk, we will discuss the following concepts: 1. KSQL Architecture 2. KSQL Use Cases 3. Performance Considerations 4. When to KSQL and When to Not 5. Introduce KStreams What this talk is: You will understand the architecture and the power of the KSQL continuous query engine and when to use it successfully. What this talk is not: An intensive KStreams talk – but you will get enough under your belt to go forth and learn more about Stream Processing overall.

Confluent Workshop Series: ksqlDB로 스트리밍 앱 빌드confluent

How to Write Great Kafka Connectorsconfluent

Watch this webcast here: https://www.confluent.io/online-talks/write-great-kafka-connectors/ Apache Kafka with all its simple but needed offerings has left deep footprints in the software industry. And, with the ever-growing and maturing Kafka ecosystem, Kafka Connect allows us to focus on the data transformation rather than handling Kafka nitty-gritty details (which is mandatory if someone is using Kafka’s Producer/Consumer APIs). Kafka Connect in contrast, provides developers and operators a simple way of accessing, transforming and delivering data to connect an organization’s applications with their event streaming platform in the form of connectors. Confluent's partner HashedIn Technologies has created many Kafka connectors. In this online talk, HashedIn shares their best practices on how to write great ones. HashedIn will cover: -In-field best practices for writing great Connectors, including both Sink Connectors and Source Connectors that transform and move data in and out of diverse external systems like AWS SQS, AWS S3, Firebase, Hadoop File System, InfluxDB, JDBC, Prometheus, Salesforce and Windows event logging. -How to unlock the true potential of the Confluent Platform and move petabytes of data each day with the best possible Kafka Connector promises including exactly/at-least-once message delivery, retry mechanism, restart/rebalance behavior and ordering guarantees.

Streaming all over the world Real life use cases with Kafka Streamsconfluent

This document discusses using Apache Kafka Streams for stream processing. It begins with an overview of Apache Kafka and Kafka Streams. It then presents several real-life use cases that have been implemented with Kafka Streams, including data conversions from XML to Avro, stream-table joins for event propagation, duplicate elimination, and detecting absence of events. The document concludes with recommendations for developing and operating Kafka Streams applications.

What's New in Confluent Platform 5.5confluent

Watch this webcast here: https://www.confluent.io/online-talks/whats-new-in-confluent-platform-55/ Join the Confluent Product Marketing team as we provide an overview of Confluent Platform 5.5, which makes Apache Kafka and event streaming more broadly accessible to developers with enhancements to data compatibility, multi-language development, and ksqlDB. Building an event-driven architecture with Apache Kafka allows you to transition from traditional silos and monolithic applications to modern microservices and event streaming applications. With these benefits has come an increased demand for Kafka developers from a wide range of industries. The Dice Tech Salary Report recently ranked Kafka as the highest-paid technological skill of 2019, a year removed from ranking it second. With Confluent Platform 5.5, we are making it even simpler for developers to connect to Kafka and start building event streaming applications, regardless of their preferred programming languages or the underlying data formats used in their applications. This session will cover the key features of this latest release, including: -Support for Protobuf and JSON schemas in Confluent Schema Registry and throughout our entire platform -Exactly once semantics for non-Java clients -Admin functions in REST Proxy (preview) -ksqlDB 0.7 and ksqlDB Flow View in Confluent Control Center

Westpac AU - Confluent Schema Registryconfluent

This document discusses using schema validation and a schema registry to ensure compatibility when data is serialized and transmitted between multiple applications and data sources. It introduces common challenges like different data formats and schemas causing issues. It then explains how tools like Avro schemas and a schema registry can define data contracts and validate messages to solve compatibility problems at scale. The document also considers how these tools integrate into Kafka pipelines and addresses questions around their usage across many developers and moving to the cloud.

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...confluent

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

Data integration with Apache Kafkaconfluent

A stream processing platform is not an island unto itself; it must be connected to all of your existing data systems, applications, and sources. In this talk we will provide different options for integrating systems and applications with Apache Kafka, with a focus on the Kafka Connect framework and the ecosystem of Kafka connectors. We will discuss the intended use cases for Kafka Connect and share our experience and best practices for building large-scale data pipelines using Apache Kafka.

Confluent Developer Trainingconfluent

This three-day course teaches developers how to build applications that can publish and subscribe to data from an Apache Kafka cluster. Students will learn Kafka concepts and components, how to use Kafka and Confluent APIs, and how to develop Kafka producers, consumers, and streams applications. The hands-on course covers using Kafka tools, writing producers and consumers, ingesting data with Kafka Connect, and more. It is designed for developers who need to interact with Kafka as a data source or destination.

Uber: Kafka Consumer Proxyconfluent

Haitao Zhang, Uber, Software Engineer + Yang Yang, Uber, Senior Software Engineer Kafka Consumer Proxy is a forwarding proxy that consumes messages from Kafka and dispatches them to a user registered gRPC service endpoint. With Kafka Consumer Proxy, the experience of consuming messages from Apache Kafka for pub-sub use cases is as seamless and user-friendly as receiving (g)RPC requests. In this talk, we will share (1) the motivation for building this service, (2) the high-level architecture, (3) the mechanisms we designed to achieve high availability, scalability, and reliability, and (4) the current adoption status. https://www.meetup.com/KafkaBayArea/events/273834934/

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...HostedbyConfluent

This document summarizes Activision Data's transition from a batch data pipeline to a real-time streaming data pipeline using Apache Kafka and Kafka Streams. Some key points: - The new pipeline ingests, processes, and stores game telemetry data from over 200k messages per second and over 5PB of data across 9 years of games. - Kafka Streams is used to transform the raw streaming data through multiple microservices with low 10-second end-to-end latency, compared to 6-24 hours previously. - Kafka Connect integrates the streaming data with data stores like AWS S3, Cassandra, and Elasticsearch. - The new pipeline provides real-time and historical access to structured

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020confluent

Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us. Things I will be sharing as part of the talk: 1. Establishing the use case/problem we were trying to solve (why we needed mTLS) 2. Building a Certificate Authority with open source tools (with self-signed Root CA) 3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers 4. Hot reloading regenerated certificates on Kafka brokers without downtime 5. What we built to rotate the self-signed root CA without downtime as well across the board 6. Monitoring and alerts on TTL of certificates 7. Performance impact of using TLS (along with why TLS affects kafka’s performance) 8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier 9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...confluent

This document discusses building microservices for data streaming and processing using Spring Cloud and Kafka. It provides an overview of Spring Cloud Stream and how it can be used to build event-driven microservices that connect to Kafka. It also discusses how Spring Cloud Data Flow can be used to orchestrate and deploy streaming applications and topologies. The document includes code samples of building a basic Kafka Streams processor application using Spring Cloud Stream and deploying it as part of a streaming data flow. It concludes with proposing a demonstration of these techniques.

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platformconfluent

Many enterprises have a large technical debt in legacy applications hosted in on-premises data centers. There is a strong desire to modernize and move to a cloud-based infrastructure, but the world won’t stop for you to transition. Existing applications need to be supported and enhanced; data from legacy platforms is required to make decisions that drive the business. On the other hand, data from cloud-based applications does not exist in a vacuum. Legacy applications need access to these cloud data sources and vice versa. Can an enterprise have it both ways? Can new applications be built in the cloud while existing applications are maintained in a private data center? Monsanto has adopted a cloud-first mentality—today most new development is focused on the cloud. However, this transition did not happen overnight. Chrix Finne and Bob Lehmann share their experience building and implementing a Kafka-based cross-data-center streaming platform to facilitate the move to the cloud—in the process, kick-starting Monsanto’s transition from batch to stream processing. Details include an overview of the challenges involved in transitioning to the cloud and a deep dive into the cross-data-center stream platform architecture, including best practices for running this architecture in production and a summary of the benefits seen after deploying this architecture.

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Common issues with Apache Kafka® Producerconfluent

Badai Aqrandista, Confluent, Senior Technical Support Engineer This session will be about a common issue in the Kafka Producer: producer batch expiry. We will be discussing the Kafka Producer internals, its common causes, such as a slow network or small batching, and how to overcome them. We will also be sharing some examples along the way! https://www.meetup.com/apache-kafka-sydney/events/279651982/

Kafka Streams for Java enthusiastsSlim Baltagi

Kafka, Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform (Kafka Core + Kafka Connect + Kafka Streams) for building streaming data pipelines and streaming data applications. This talk, that I gave at the Chicago Java Users Group (CJUG) on June 8th 2017, is mainly focusing on Kafka Streams, a lightweight open source Java library for building stream processing applications on top of Kafka using Kafka topics as input/output. You will learn more about the following: 1. Apache Kafka: a Streaming Data Platform 2. Overview of Kafka Streams: Before Kafka Streams? What is Kafka Streams? Why Kafka Streams? What are Kafka Streams key concepts? Kafka Streams APIs and code examples? 3. Writing, deploying and running your first Kafka Streams application 4. Code and Demo of an end-to-end Kafka-based Streaming Data Application 5. Where to go from here?

Understanding Apache Kafka® Latency at Scaleconfluent

Apache Kafka® in Industrial Environments confluent

3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)confluent

What's inside the black box? Using ML to tune and manage Kafka. (Matthew Stum...confluent

We use machine learning to delve deep into the internals of how systems like Kafka work. In this talk I'll dive into what variables affect performance and reliability, including previously unknown leading indicators of major performance problems, failure conditions and how to tune for specific use cases. I'll cover some of the specific methodology we use, including Bayesian optimization, and reinforcement learning. I'll also talk about our own internal infrastructure that makes heavy use of Kafka and Kubernetes to deliver real-time predictions to our customers.

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...confluent

Vahid Hashemian and Ambud Sharma from Pinterest discuss Kafka operations at their company. Pinterest uses over 50 Kafka clusters with 2,500+ brokers to ingest 20+ GB/s of data and output 50+ GB/s. They faced challenges around performance, costs, and dynamic partitioning. To address these, Pinterest developed automation tools like Orion to manage clusters and topics, upgraded all clusters to Kafka 2.3.1+, and learned lessons around testing upgrade versions and backward compatibility. Going forward, they aim to improve interoperability, scaling, efficiency, and reliability.

Intro to AsyncAPIconfluent

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...confluent

What do you do when you've two different technologies on the upstream and the downstream that are both rapidly being adopted industrywide? How do you bridge them scalably and robustly? At Wework, the upstream data was being brokered by Kafka and the downstream consumers were highly scalable gRPC services. While Kafka was capable of efficiently channeling incoming events in near real-time from a variety of sensors that were used in select Wework spaces, the downstream gRPC services that were user-facing were exceptionally good at serving requests in a concurrent and robust manner. This was a formidable combination, if only there was a way to effectively bridge these two in an optimized way. Luckily, sink Connectors came to the rescue. However, there weren't any for gRPC sinks! So we wrote one. In this talk, we will briefly focus on the advantages of using Connectors, creating new Connectors, and specifically spend time on gRPC sink Connector and its impact on Wework's data pipeline.

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 confluent

Namely is a late-stage startup that builds HR, Payroll and Benefits software for mid-sized businesses. Over the years, we've ended up with a number of monolithic and legacy applications covering overlapping domain concepts, which has limited our ability to deliver new and innovative features to our customers. We need a way to get our data out of the monoliths to decouple our systems and increase our velocity. We've chosen Kafka as our way to liberate our data in a reliable, scalable and maintainable way. This talk covers specific examples of successes and missteps in our move to Kafka as the backbone of our architecture. It then looks to the future - where we are trying to go, and how we plan on getting, both from the short term and long term perspectives. Key Takeaways: - Successful and unsuccessful approaches to gradually introducing Kafka to a large organization in a way that meets the short and long term needs of the business. - Successful and unsuccessful patterns for using Kafka. - Pragmaticism versus purisim: Building Kafka-first systems, and migrating legacy systems to Kafka with Debezium. - Combining event driven systems with RPC based systems. Observability, alerting and testing. - Actionable steps that you can take to your organization to help drive adoption.

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...HostedbyConfluent

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. Kafka was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

Building an Event-oriented Data Platform with Kafka, Eric Sammer confluent

While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. Many organizations understand the use cases around their data – fraud detection, quality of service and technical operations, user behavior analysis, for example – but are not necessarily data infrastructure experts. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes an hour of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. Attendees will leave this session knowing not just which open source projects go into a system such as this, but how they work together, what tradeoffs and decisions need to be addressed, and how to present a single general purpose data platform to multiple applications. This session should be attended by data infrastructure engineers and architects planning, building, or maintaining similar systems.

ksqlDB: A Stream-Relational Database Systemconfluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

Devops architectureOjasvi Jagtap

Devops architecture involves three main categories of infrastructure: IT infrastructure (version control, issue tracking, etc.), build infrastructure (build servers with access to source code), and test infrastructure (deployment, acceptance, and functional testing). Continuous integration involves automating the integration of code changes, while continuous delivery ensures code is always releasable but actual deployment is manual. Continuous deployment automates deployment so that any code passing tests is immediately deployed to production. The document discusses infrastructure hosting options, automation approaches, common CI/CD workflows, and provides examples of low and medium-cost devops tooling setups using open source and proprietary software.

KCD Munich - Cloud Native Platform Dilemma - Turning it into an OpportunityAndreas Grabner

This talk was given at KCD Munich - July 17 2023 Abstract “Kubernetes is a platform for building platforms. It’s a better place to start: not the endgame”, tweeted by Kelsey Hightower in November 2017. 6 years later the Cloud Native Community is faced with 159 different CNCF projects to choose from. Entering CNCF can be overwhelming! Cloud Native Platform Engineering with white papers, best practices and reference architectures are here to convert this dilemma into an opportunity. Internal Developer Platforms (IDP) are being built as we speak enabling organizations to harness the power of Kubernetes as a self-service platform. Join this talk with Andreas Grabner, CNCF Ambassador, and get some insights on tooling, use cases and best practices so we can all fulfill the idea that Kelsey put out years ago.

More Related Content

What's hot (20)

Confluent Developer Trainingconfluent

Uber: Kafka Consumer Proxyconfluent

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...HostedbyConfluent

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020confluent

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...confluent

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platformconfluent

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Common issues with Apache Kafka® Producerconfluent

Kafka Streams for Java enthusiastsSlim Baltagi

Understanding Apache Kafka® Latency at Scaleconfluent

Apache Kafka® in Industrial Environments confluent

3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)confluent

What's inside the black box? Using ML to tune and manage Kafka. (Matthew Stum...confluent

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...confluent

Intro to AsyncAPIconfluent

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...confluent

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 confluent

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...HostedbyConfluent

Building an Event-oriented Data Platform with Kafka, Eric Sammer confluent

ksqlDB: A Stream-Relational Database Systemconfluent

Confluent Developer Trainingconfluent

Uber: Kafka Consumer Proxyconfluent

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...HostedbyConfluent

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020confluent

Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...confluent

Stream Me Up, Scotty: Transitioning to the Cloud Using a Streaming Data Platformconfluent

Westpac Bank Tech Talk 1: Dive into Apache Kafkaconfluent

Common issues with Apache Kafka® Producerconfluent

Kafka Streams for Java enthusiastsSlim Baltagi

Understanding Apache Kafka® Latency at Scaleconfluent

Apache Kafka® in Industrial Environments confluent

3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)confluent

What's inside the black box? Using ML to tune and manage Kafka. (Matthew Stum...confluent

Organic Growth and A Good Night Sleep: Effective Kafka Operations at Pinteres...confluent

Intro to AsyncAPIconfluent

Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...confluent

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 confluent

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...HostedbyConfluent

Building an Event-oriented Data Platform with Kafka, Eric Sammer confluent

ksqlDB: A Stream-Relational Database Systemconfluent

Similar to Leveraging Microservices and Apache Kafka to Scale Developer Productivity (20)

Devops architectureOjasvi Jagtap

KCD Munich - Cloud Native Platform Dilemma - Turning it into an OpportunityAndreas Grabner

IBM BP Session - Multiple CLoud Paks and Cloud Paks Foundational Services.pptxGeorg Ember

Diese Präsentation beinhaltet Erfahrungen, Empfehlungen und Planungs-Gedanken, die man beachten sollte, wenn man multiple IBM Cloud Paks auf der Container Platform OpenShift installieren / deployen möchte. Es beschreibt die Grundlagen zu "common services", auch "foundational services" genannt, die als Basis-Services die Lauffähigkeit dieser Cloud Paks auf OpenShift erläutert und wie man Cloud Paks auch logisch trennen kann auf OpenShift worker nodes über taints und node selectors.

DevOps : Integrate, Deliver and Deploy continuously with Visual Studio Team S...BAINIDA

MLOps in actionPieter de Bruin

Dev ops using JenkinsSynergetics Learning and Cloud Consulting

Continuous delivery is the process of automating the deployment of code changes to production. It involves building, testing, and deploying code changes through successive environments like integration, testing, and production. Continuous integration starts the process by automatically building and testing code changes. The release pipeline then automates deploying through environments. This finds issues early and allows for rapid deployment of code changes to production through automated testing and infrastructure provisioning.

Continuous Delivery with a PaaS ApplicationMark Rendell

Modernizing Testing as Apps Re-ArchitectDevOps.com

Applications are moving to cloud and containers to boost reliability and speed delivery to production. However, if we use the same old approaches to testing, we'll fail to achieve the benefits of cloud. But what do we really need to change? We know we need to automate tests, but how do we keep our automation assets from becoming obsolete? Automatically provisioning test environments seems close, but some parts of our applications are hard to move to cloud.

NET Aspire - NET Conf IL 2024 - Tamir Dresher.pdfTamir Dresher

Azure DevOps in ActionCallon Campbell

Microsoft recently released Azure DevOps, a set of services that help developers and IT ship software faster, and with higher quality. These services cover planning, source code, builds, deployments, and artifacts. One of the great things about Azure DevOps is that it works great for any app and on any platform regardless of frameworks. In this session, I will provide a hands on workshop guiding you through getting started with Azure Pipelines to build your application. Using continuous integration and deployment processes, you will leave with clear understanding and skills to get your applications up and running quickly in Azure DevOps and see the full benefits that CI/CD can bring to your organization.

Dev ops for cross platform mobile modeveast 12Sanjeev Sharma

Mobile Apps are not stand alone applications running on a mobile device anymore. Apps today are complex systems with back-ends hosted on clouds, with application servers, databases, API calls to external systems, and of course a powerful app running on a mobile device. Mobile App development and deployment is further complicated with todays need for supporting multiple mobile devices, with multiple OSes, multiple versions of the OSes, multiple form factors and varied network, CPU, GPU and memory specs. DevOps - the new and growing movement addresses these development and deployment challenges. The goal of DevOps is to align Dev and Ops by introducing a set of principles and practices such as continuous integration and continuous delivery. Mobile apps take the need for these practices up a level due to their inherent distributed nature. Multi-platform mobile apps need even more care in applying DevOps principles as there are multiple platforms to be targeted, each with its own requirements, quirks, and nuanced needs. This talk will introduce attendees to the basic practices of DevOps and then take a look at the DevOps challenges specific to cross-platform Mobile apps and present Best Practices to address them.

Growing Adoption of Open Source in EnterprisesWSO2

This document discusses the growing adoption of open source in enterprises. It provides an agenda that covers why open source is being adopted, key considerations for adoption, a suggested adoption roadmap, professional support offerings, the WSO2 open source platform, and leaves time for questions. The presentation then discusses the benefits of open source like innovation, cost reductions, and avoidance of vendor lock-in. It outlines WSO2's open source platform and support model to help enterprises adopt and optimize their use of open source.

20171122 aws usergrp_coretech-spn-cicd-aws-v01Scott Miao

This document discusses SPN's journey to implement CI/CD on AWS. It begins with describing SPN's original process for delivering services which involved many manual steps. It then discusses DevOps goals of faster delivery, lower failure rates, and faster recovery compared to the original process. The document outlines using AWS services like CloudFormation, OpsWorks, and Auto Scaling to implement CI/CD and automate deploying a sample analytic engine service. Lessons learned include automating as much as possible, splitting CloudFormation templates, focusing on updates without impacting SLAs, and emphasizing monitoring and testing.

Netflix Cloud Architecture and Open Sourceaspyker

DevOps Evolution - The Next Generation ?Marc Hornbeek

Simplify and Scale Enterprise Spring Apps in the Cloud | March 23, 2023VMware Tanzu

- Azure Spring Apps is a fully managed service for deploying and managing Spring Boot apps in the cloud without having to learn or manage Kubernetes. It provides auto-scaling, security, high availability, and auto-patching capabilities. - Managing software updates and security patches across multiple components like apps, dependencies, JDKs, OSes, Kubernetes, etc. is challenging due to the large volume of updates and need for testing and approvals. Azure Spring Apps reduces this burden through auto-patching which applies critical security updates automatically during scheduled maintenance windows. - Auto-patching helps customers stay ahead of security threats and vulnerabilities by proactively applying patches for exposed issues like Log4j, OpenSSL vulnerabilities,

Azure DevOps työkalut - Roundtable 14.3.2019Janne Mattila

The document is an agenda for an event discussing Azure DevOps tools and projects. The agenda includes: - Breakfast and opening from 8:30-9:00 - A presentation on Azure DevOps tools from 9:00-9:45 - A presentation on Azure PaaS projects and agile development from 9:45-10:30 - A panel discussion from 10:30 - Lunch The document provides details on the presentations and panels planned during the event.

Platform as a Runtime - PaaR QCON 2024 - FinalAviran Mordo

In this talk, Aviran will describe how http://Wix.com is pushing this trend even further to build its own Platform as a Runtime (PaaR) infrastructure that allows developers to develop faster, better with higher quality. By allowing nano deployments of different modules into a “SingleRuntime” inside a robust internal platform that handles many of the non-functional concerns developers are facing on a daily basis.

Ibm innovate ci for system zRosalind Radcliffe

This document discusses continuous integration for System z mainframe applications. It begins with an overview of DevOps and continuous integration concepts. It then discusses the IBM DevOps solution and challenges of applying DevOps to System z environments. The document focuses on how continuous integration can be implemented for System z to provide rapid feedback, automated testing in isolated environments, and higher quality code promoted between stages. It also discusses how continuous testing can be achieved through dependency virtualization to improve testing efficiency.

Salesforce DevOps Certification - Salesforce DevOps Training.pptxhimavanthvisualpath

Join Visualpath - Salesforce DevOps Training hands-on learning and real-time project experience. Salesforce DevOps Course expert trainers, with over 10 years of industry experience, ensure you gain practical skills and real-time examples, and in-depth learning, resume preparation, technical doubt clarification. Our Salesforce DevOps Online Training Accessible globally in regions like the USA, UK, Canada, Dubai, and Australia. For more info, call +91-7032290546. Key Points: yaml, git, bit bucket, autorabit, shell scripting, ant migration WhatsApp: https://wa.me/c/917032290546 Visit: https://www.visualpath.in/online-salesforce-devops-training.html Visit our Blog:https://visualpathblogs.com/category/salesforce-devops-with-copado/

Devops architectureOjasvi Jagtap

KCD Munich - Cloud Native Platform Dilemma - Turning it into an OpportunityAndreas Grabner

IBM BP Session - Multiple CLoud Paks and Cloud Paks Foundational Services.pptxGeorg Ember

DevOps : Integrate, Deliver and Deploy continuously with Visual Studio Team S...BAINIDA

MLOps in actionPieter de Bruin

Dev ops using JenkinsSynergetics Learning and Cloud Consulting

Continuous Delivery with a PaaS ApplicationMark Rendell

Modernizing Testing as Apps Re-ArchitectDevOps.com

NET Aspire - NET Conf IL 2024 - Tamir Dresher.pdfTamir Dresher

Azure DevOps in ActionCallon Campbell

Dev ops for cross platform mobile modeveast 12Sanjeev Sharma

Growing Adoption of Open Source in EnterprisesWSO2

20171122 aws usergrp_coretech-spn-cicd-aws-v01Scott Miao

Netflix Cloud Architecture and Open Sourceaspyker

DevOps Evolution - The Next Generation ?Marc Hornbeek

Simplify and Scale Enterprise Spring Apps in the Cloud | March 23, 2023VMware Tanzu

Azure DevOps työkalut - Roundtable 14.3.2019Janne Mattila

Platform as a Runtime - PaaR QCON 2024 - FinalAviran Mordo

Ibm innovate ci for system zRosalind Radcliffe

Salesforce DevOps Certification - Salesforce DevOps Training.pptxhimavanthvisualpath

More from confluent (20)

Webinar Think Right - Shift Left - 19-03-2025.pptxconfluent

Migration, backup and restore made easy using Kannikaconfluent

In this presentation, you’ll discover how easily you can migrate data from any Kafka-compatible event hub to Confluent using Kannika’s intuitive self-service interface. We’ll guide you through the process, showing how the same approach can be applied to define specific event data sets and effortlessly spin up secure environments for demos, testing, or other purposes. You’ll also learn how to back up event data in just a few steps by transferring compressed data to the cloud storage location of your choice. In addition, we’ll demonstrate how to restore filtered datasets of topics, ensuring quick recovery and maintaining business continuity when needed.

Five Things You Need to Know About Data Streaming in 2025confluent

Topics that Peter covers: Tapping into the Potential of Data Products: Data drives some of today's most important business use cases. Data products enable instant access to reliable and trustworthy data by eliminating the data mess created by point-to-point connections. The Need to Tap into 'Quick Thinking': The C-level has to reorient itself so it doesn't become the bottleneck to adaptability in a data-driven world. Nine in 10 (90%) business leaders say they must now react in real-time. Learn what you can do to provide executive access to real-time data to enable 'Quick Thinking.' Rise Above Data Hurdles: Discover how to enforce governance at data production. Reestablishing trustworthiness later is almost always harder, so investing in data tools that solve business problems rather than add to them is essential. Paradigm to Shift Left: Shift Left is a new paradigm for processing and governing data at any scale, complexity, and latency. Shift Left moves the processing and governance of data closer to the source, enabling organisations to build their data once, build it right and reuse it anywhere within moments of its creation. The Need for a Strategic View: The positive correlation between data streaming maturity and significant business returns underscores the importance of a long-term, strategic view of data streaming investments. It also highlights the value of advancing beyond initial, siloed use cases to a more integrated approach that leverages data streaming across the enterprise.

Data in Motion Tour Seoul 2024 - Keynoteconfluent

Data in Motion Tour Seoul 2024 - Roadmap Democonfluent

From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...confluent

In this presentation, we’ll demonstrate how Confluent and Lightstreamer come together to tackle the last-mile challenge of extending your Kafka architecture to web and mobile platforms. Learn how to effortlessly build real-time web applications within minutes, subscribing to Kafka topics directly from your web pages, with unmatched low latency and high scalability. Explore how Confluent's leading Kafka platform and Lightstreamer's intelligent proxy work seamlessly to bridge Kafka with the internet frontier, delivering data in real-time.

Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent

Data in Motion Tour 2024 Riyadh, Saudi Arabiaconfluent

Data streaming platforms are becoming increasingly important in today’s fast-paced world. From retail giants who need to monitor inventory levels to ensure stores never run out of items, to new-age, innovative banks who are building out-of-the-box banking solutions for traditional retail banks, data streaming platforms are at the centre, powering these workflows. Data streaming platforms connect all your applications, systems, and teams with a shared view of the most up-to-date, real-time data. From Gen AI, stream governance to stream processing - it’s these cutting edge developments that will be featured during the day.

Build a Real-Time Decision Support Application for Financial Market Traders w...confluent

Strumenti e Strategie di Stream Governance con Confluent Platformconfluent

Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeksconfluent

As businesses strive to stay at the forefront of innovation, the ability to quickly develop scalable Generative AI (GenAI) applications is essential. Join us for an exclusive webinar featuring MIA Platform, MongoDB, and Confluent, where you'll learn how to compose GenAI apps with real-time data integration in a fraction of the time. Discover how these three powerful platforms work together to ensure applications remain responsive, relevant, and adaptive to user preferences and contextual changes. Our experts will guide you through leveraging MIA Platform's microservices architecture and low-code development, MongoDB's flexibility, and Confluent's stream processing capabilities. Experience live demonstrations and practical insights that will transform your approach to AI-driven app development, enabling you to accelerate your development process from weeks to mere minutes. Don't miss this opportunity to keep your business at the cutting edge.

Building Real-Time Gen AI Applications with SingleStore and Confluentconfluent

Unlocking value with event-driven architecture by Confluentconfluent

Sfrutta il potere dello streaming di dati in tempo reale e dei microservizi basati su eventi per il futuro di Sky con Confluent e Kafka®. In questo tech talk esploreremo le potenzialità di Confluent e Apache Kafka® per rivoluzionare l'architettura aziendale e sbloccare nuove opportunità di business. Ne approfondiremo i concetti chiave, guidandoti nella creazione di applicazioni scalabili, resilienti e fruibili in tempo reale per lo streaming di dati. Scoprirai come costruire microservizi basati su eventi con Confluent, sfruttando i vantaggi di un'architettura moderna e reattiva. Il talk presenterà inoltre casi d'uso reali di Confluent e Kafka®, dimostrando come queste tecnologie possano ottimizzare i processi aziendali e generare valore concreto.

Il Data Streaming per un’AI real-time di nuova generazioneconfluent

Per costruire applicazioni di AI affidabili, sicure e governate occorre una base dati in tempo reale altrettanto solida. Ancor più quando ci troviamo a gestire ingenti flussi di dati in continuo movimento. Come arrivarci? Affidati a una vera piattaforma di data streaming che ti permetta di scalare e creare rapidamente applicazioni di AI in tempo reale partendo da dati affidabili. Scopri di più! Non perdere il nostro prossimo webinar durante il quale avremo l’occasione di: • Esplorare il paradigma della GenAI e di come questa nuova tecnnologia sta rimodellando il panorama aziendale, rispondendo alla necessità di offrire un contesto e soluzioni in tempo reale che soddisfino le esigenze della tua azienda. • Approfondire le incertezze del panorama dell'AI in evoluzione e l'importanza cruciale del data streaming e dell'elaborazione dati. • Vedere in dettaglio l'architettura in continua evoluzione e il ruolo chiave di Kafka e Confluent nelle applicazioni di AI. • Analizzare i vantaggi di una piattaforma di streaming dei dati come Confluent nel collegare l'eredità legacy e la GenAI, facilitando lo sviluppo e l’utilizzo di AI predittive e generative.

Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent

As businesses strive to remain at the cutting edge of innovation, the demand for scalable and up-to-date conversational AI solutions has become paramount. Generative AI (GenAI) chatbots that seamlessly integrate into our daily lives and adapt to the ever-evolving nuances of human interaction are crucial. Real-time data plays a pivotal role in ensuring the responsiveness and relevance of these chatbots, empowering them to stay abreast of the latest trends, user preferences, and contextual information.

Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent

Building API data products on top of your real-time data infrastructureconfluent

This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products. You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering. You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks. Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!

Speed Wins: From Kafka to APIs in Minutesconfluent

Evolving Data Governance for the Real-time Streaming and AI Eraconfluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent