This document summarizes a microservices meetup hosted by @mosa_siru. Key points include:
1. @mosa_siru is an engineer at DeNA and CTO of Gunosy.
2. The meetup covered Gunosy's architecture with over 45 GitHub repositories, 30 stacks, 10 Go APIs, and 10 Python batch processes using AWS services like Kinesis, Lambda, SQS and API Gateway.
3. Challenges discussed were managing 30 microservices, ensuring API latency below 50ms across availability zones, and handling 10 requests per second with nginx load balancing across 20 servers.
This document discusses reliability guarantees in Apache Kafka. It explains that Kafka provides reliability through replication of data across multiple brokers. It describes concepts like in-sync replicas, unclean leader election, and how to configure replication factor and minimum in-sync replicas. The document also covers best practices for producers like setting acks to all, and for consumers like committing offsets manually and handling rebalances. It emphasizes the importance of monitoring for errors, lag, and data reconciliation to ensure reliability.
The document discusses Apache OpenWhisk, a serverless computing platform or Function as a Service (FaaS) that allows running code in response to events. It provides an overview of OpenWhisk architecture including the controller, invoker, and provider components. Examples of using OpenWhisk with Yahoo! Japan and for FaaS are described. Monitoring and developer tools for OpenWhisk are also mentioned.
Migrating Airflow-based Apache Spark Jobs to Kubernetes – the Native WayDatabricks
At Nielsen Identity, we use Apache Spark to process 10’s of TBs of data, running on AWS EMR. We started at a point where Spark was not even supported out-of-the-box by EMR, and today we’re spinning-up clusters with 1000’s of nodes on a daily basis, orchestrated by Airflow. A few months ago, we embarked on a journey to evaluate the option of using Kubernetes as our Spark infrastructure, mainly to reduce operational costs and improve stability (as we heavily rely on Spot Instances for our clusters). To allow us to achieve those goals, we combined the open-sourced GCP Spark-on-K8s operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) with a native Airflow integration we developed and recently contributed back to the Airflow project (https://issues.apache.org/jira/browse/AIRFLOW-6542). Finally, we were able to migrate our existing Airflow DAGs, with minimal changes, from AWS EMR to K8s.
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Flink Forward
Flink Forward San Francisco 2022.
Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way.
by
Jeff Chao
This document compares Apache Kafka and AWS Kinesis for message streaming. It outlines that Kafka is an open source publish-subscribe messaging system designed as a distributed commit log, while Kinesis provides streaming data services. It also notes some key differences like Kafka typically handling over 8000 messages/second while Kinesis can handle under 100 messages/second.
2018/11/22に「サーバーサイドエンジニアあるある」というテーマでLTをしました。(会場提供:アンドファクトリー様)
https://andfactory.connpass.com/event/106362/
A Lightning Talk at $ Factory in Shibuya on 22/11/2018
The theme was "Typical things for server side programmers".
This document provides an overview of patterns for scalability, availability, and stability in distributed systems. It discusses general recommendations like immutability and referential transparency. It covers scalability trade-offs around performance vs scalability, latency vs throughput, and availability vs consistency. It then describes various patterns for scalability including managing state through partitioning, caching, sharding databases, and using distributed caching. It also covers patterns for managing behavior through event-driven architecture, compute grids, load balancing, and parallel computing. Availability patterns like fail-over, replication, and fault tolerance are discussed. The document provides examples of popular technologies that implement many of these patterns.
Stream Processing with Flink and Stream Sharingconfluent
Stream Processing with Apache Flink in the Cloud and Stream Sharing
Kai Waehner discusses stream processing use cases with Apache Flink and introduces Confluent's plans to offer a serverless Apache Flink service in Confluent Cloud. He also discusses Confluent Stream Sharing, which allows for easy and secure sharing of Kafka topics across organizations in real-time.
- Apache Spark is an open-source cluster computing framework for large-scale data processing. It was originally developed at the University of California, Berkeley in 2009 and is used for distributed tasks like data mining, streaming and machine learning.
- Spark utilizes in-memory computing to optimize performance. It keeps data in memory across tasks to allow for faster analytics compared to disk-based computing. Spark also supports caching data in memory to optimize repeated computations.
- Proper configuration of Spark's memory options is important to avoid out of memory errors. Options like storage fraction, execution fraction, on-heap memory size and off-heap memory size control how Spark allocates and uses memory across executors.
Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop.
It's also enabling many real-time system frameworks and use cases.
Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka
in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API.
Also talk about the best practices involved in running a producer/consumer.
In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects.
We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing
Kafka ACLs and monitoring Consumer offsets.
Apache Superset - open source data exploration and visualization (Conclusion ...Lucas Jellema
Introducing Apache Superset - an open source platform for data exploration, visualization and analysis - co-starring Trino and Steampipe for providing SQL access to many non-SQL data sources.
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022HostedbyConfluent
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
An instant world requires instant decisions at scale. This includes the ability to digest and react to changes in real-time. Thus, event logs such as Apache Kafka can be found in almost every architecture, while databases and similar systems still provide the foundation. Change Data Capture (CDC) has become popular for propagating changes. Nevertheless, integrating all these systems, which often have slightly different semantics, can be a challenge.
In this talk, we highlight what it means for Apache Flink to be a general data processor that acts as a data integration hub. Looking under the hood, we demonstrate Flink's SQL engine as a changelog processor that ships with an ecosystem tailored to processing CDC data and maintaining materialized views. We will discuss the semantics of different data sources and how to perform joins or stream enrichment between them. This talk illustrates how Flink can be used with systems such as Kafka (for upsert logging), Debezium, JDBC, and others.
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
AWS Data Lake / Lake House + Confluent Cloud for Serverless Apache Kafka. Learn about use cases, architectures, and features.
Data must be continuously collected, processed, and reactively used in applications across the entire enterprise - some in real time, some in batch mode. In other words: As an enterprise becomes increasingly software-defined, it needs a data platform designed primarily for "data in motion" rather than "data at rest."
Apache Kafka is now mainstream when it comes to data in motion! The Kafka API has become the de facto standard for event-driven architectures and event streaming. Unfortunately, the cost of running it yourself is very often too expensive when you add factors like scaling, administration, support, security, creating connectors...and everything else that goes with it. Resources in enterprises are scarce: this applies to both the best team members and the budget.
The cloud - as we all know - offers the perfect solution to such challenges.
Most likely, fully-managed cloud services such as AWS S3, DynamoDB or Redshift are already in use. Now it is time to implement "fully-managed" for Kafka as well - with Confluent Cloud on AWS.
Building a central integration layer that doesn't care where or how much data is coming from.
Implementing scalable data stream processing to gain real-time insights
Leveraging fully managed connectors (like S3, Redshift, Kinesis, MongoDB Atlas & more) to quickly access data
Confluent Cloud in action? Let's show how ao.com made it happen!
Translated with www.DeepL.com/Translator (free version)
Rethinking Geo-replication for the Cloud | Luke Knepper, ConfluentHostedbyConfluent
This document discusses rethinking geo-replication for the cloud using cluster linking. Cluster linking allows linking Kafka clusters across different cloud regions and on-premises environments. This provides a simpler way to implement geo-replication compared to existing tools. Cluster linking also enables multi-region architectures with consistent offsets and disaster recovery capabilities by failing over to a secondary region. Demostrations show how cluster linking can be used for multi-region deployments and disaster recovery failovers.
How to Build Streaming Apps with Confluent IIconfluent
In this interactive session, you’ll access a lab environment that shows you how to build Streaming Applications on top of Kafka, leveraging Confluent's modern tooling.
This is your exclusive opportunity to hear from the thought leaders of Apache Kafka on how event streaming enables you to leverage real-time data processing, with an easy-to-use, yet powerful interactive interface for stream processing, without the need to write code.
The document discusses Apache OpenWhisk, a serverless computing platform or Function as a Service (FaaS) that allows running code in response to events. It provides an overview of OpenWhisk architecture including the controller, invoker, and provider components. Examples of using OpenWhisk with Yahoo! Japan and for FaaS are described. Monitoring and developer tools for OpenWhisk are also mentioned.
Migrating Airflow-based Apache Spark Jobs to Kubernetes – the Native WayDatabricks
At Nielsen Identity, we use Apache Spark to process 10’s of TBs of data, running on AWS EMR. We started at a point where Spark was not even supported out-of-the-box by EMR, and today we’re spinning-up clusters with 1000’s of nodes on a daily basis, orchestrated by Airflow. A few months ago, we embarked on a journey to evaluate the option of using Kubernetes as our Spark infrastructure, mainly to reduce operational costs and improve stability (as we heavily rely on Spot Instances for our clusters). To allow us to achieve those goals, we combined the open-sourced GCP Spark-on-K8s operator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator) with a native Airflow integration we developed and recently contributed back to the Airflow project (https://issues.apache.org/jira/browse/AIRFLOW-6542). Finally, we were able to migrate our existing Airflow DAGs, with minimal changes, from AWS EMR to K8s.
Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Cap...Flink Forward
Flink Forward San Francisco 2022.
Being in the payments space, Stripe requires strict correctness and freshness guarantees. We rely on Flink as the natural solution for delivering on this in support of our Change Data Capture (CDC) infrastructure. We heavily rely on CDC as a tool for capturing data change streams from our databases without critically impacting database reliability, scalability, and maintainability. Data derived from these streams is used broadly across the business and powers many of our critical financial reporting systems totalling over $640 Billion in payment volume annually. We use many components of Flink’s flexible DataStream API to perform aggregations and abstract away the complexities of stream processing from our downstreams. In this talk, we’ll walk through our experience from the very beginning to what we have in production today. We’ll share stories around the technical details and trade-offs we encountered along the way.
by
Jeff Chao
This document compares Apache Kafka and AWS Kinesis for message streaming. It outlines that Kafka is an open source publish-subscribe messaging system designed as a distributed commit log, while Kinesis provides streaming data services. It also notes some key differences like Kafka typically handling over 8000 messages/second while Kinesis can handle under 100 messages/second.
2018/11/22に「サーバーサイドエンジニアあるある」というテーマでLTをしました。(会場提供:アンドファクトリー様)
https://andfactory.connpass.com/event/106362/
A Lightning Talk at $ Factory in Shibuya on 22/11/2018
The theme was "Typical things for server side programmers".
This document provides an overview of patterns for scalability, availability, and stability in distributed systems. It discusses general recommendations like immutability and referential transparency. It covers scalability trade-offs around performance vs scalability, latency vs throughput, and availability vs consistency. It then describes various patterns for scalability including managing state through partitioning, caching, sharding databases, and using distributed caching. It also covers patterns for managing behavior through event-driven architecture, compute grids, load balancing, and parallel computing. Availability patterns like fail-over, replication, and fault tolerance are discussed. The document provides examples of popular technologies that implement many of these patterns.
Stream Processing with Flink and Stream Sharingconfluent
Stream Processing with Apache Flink in the Cloud and Stream Sharing
Kai Waehner discusses stream processing use cases with Apache Flink and introduces Confluent's plans to offer a serverless Apache Flink service in Confluent Cloud. He also discusses Confluent Stream Sharing, which allows for easy and secure sharing of Kafka topics across organizations in real-time.
- Apache Spark is an open-source cluster computing framework for large-scale data processing. It was originally developed at the University of California, Berkeley in 2009 and is used for distributed tasks like data mining, streaming and machine learning.
- Spark utilizes in-memory computing to optimize performance. It keeps data in memory across tasks to allow for faster analytics compared to disk-based computing. Spark also supports caching data in memory to optimize repeated computations.
- Proper configuration of Spark's memory options is important to avoid out of memory errors. Options like storage fraction, execution fraction, on-heap memory size and off-heap memory size control how Spark allocates and uses memory across executors.
Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop.
It's also enabling many real-time system frameworks and use cases.
Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka
in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API.
Also talk about the best practices involved in running a producer/consumer.
In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects.
We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing
Kafka ACLs and monitoring Consumer offsets.
Apache Superset - open source data exploration and visualization (Conclusion ...Lucas Jellema
Introducing Apache Superset - an open source platform for data exploration, visualization and analysis - co-starring Trino and Steampipe for providing SQL access to many non-SQL data sources.
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022HostedbyConfluent
CDC Stream Processing With Apache Flink With Timo Walther | Current 2022
An instant world requires instant decisions at scale. This includes the ability to digest and react to changes in real-time. Thus, event logs such as Apache Kafka can be found in almost every architecture, while databases and similar systems still provide the foundation. Change Data Capture (CDC) has become popular for propagating changes. Nevertheless, integrating all these systems, which often have slightly different semantics, can be a challenge.
In this talk, we highlight what it means for Apache Flink to be a general data processor that acts as a data integration hub. Looking under the hood, we demonstrate Flink's SQL engine as a changelog processor that ships with an ecosystem tailored to processing CDC data and maintaining materialized views. We will discuss the semantics of different data sources and how to perform joins or stream enrichment between them. This talk illustrates how Flink can be used with systems such as Kafka (for upsert logging), Debezium, JDBC, and others.
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
AWS Data Lake / Lake House + Confluent Cloud for Serverless Apache Kafka. Learn about use cases, architectures, and features.
Data must be continuously collected, processed, and reactively used in applications across the entire enterprise - some in real time, some in batch mode. In other words: As an enterprise becomes increasingly software-defined, it needs a data platform designed primarily for "data in motion" rather than "data at rest."
Apache Kafka is now mainstream when it comes to data in motion! The Kafka API has become the de facto standard for event-driven architectures and event streaming. Unfortunately, the cost of running it yourself is very often too expensive when you add factors like scaling, administration, support, security, creating connectors...and everything else that goes with it. Resources in enterprises are scarce: this applies to both the best team members and the budget.
The cloud - as we all know - offers the perfect solution to such challenges.
Most likely, fully-managed cloud services such as AWS S3, DynamoDB or Redshift are already in use. Now it is time to implement "fully-managed" for Kafka as well - with Confluent Cloud on AWS.
Building a central integration layer that doesn't care where or how much data is coming from.
Implementing scalable data stream processing to gain real-time insights
Leveraging fully managed connectors (like S3, Redshift, Kinesis, MongoDB Atlas & more) to quickly access data
Confluent Cloud in action? Let's show how ao.com made it happen!
Translated with www.DeepL.com/Translator (free version)
Rethinking Geo-replication for the Cloud | Luke Knepper, ConfluentHostedbyConfluent
This document discusses rethinking geo-replication for the cloud using cluster linking. Cluster linking allows linking Kafka clusters across different cloud regions and on-premises environments. This provides a simpler way to implement geo-replication compared to existing tools. Cluster linking also enables multi-region architectures with consistent offsets and disaster recovery capabilities by failing over to a secondary region. Demostrations show how cluster linking can be used for multi-region deployments and disaster recovery failovers.
How to Build Streaming Apps with Confluent IIconfluent
In this interactive session, you’ll access a lab environment that shows you how to build Streaming Applications on top of Kafka, leveraging Confluent's modern tooling.
This is your exclusive opportunity to hear from the thought leaders of Apache Kafka on how event streaming enables you to leverage real-time data processing, with an easy-to-use, yet powerful interactive interface for stream processing, without the need to write code.
This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.
Building API data products on top of your real-time data infrastructureconfluent
This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products.
You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering.
You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks.
Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...HostedbyConfluent
We will demonstrate how easy it is to use Confluent Cloud as the data source of your Beam pipelines. You will learn how to process the information that comes from Confluent Cloud in real time, make transformations on such information and feed it back to your Kafka topics and other parts of your architecture.
Build a Real-Time Decision Support Application for Financial Market Traders w...confluent
Quix's intuitive visual programming interface and extensive library of pre-built components make it easy to build these applications without complex coding. Experience how this dynamic duo accelerates the development and deployment of your trading strategies, empowering you to make more informed decisions with real-time data!
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent
As businesses strive to remain at the cutting edge of innovation, the demand for scalable and up-to-date conversational AI solutions has become paramount. Generative AI (GenAI) chatbots that seamlessly integrate into our daily lives and adapt to the ever-evolving nuances of human interaction are crucial. Real-time data plays a pivotal role in ensuring the responsiveness and relevance of these chatbots, empowering them to stay abreast of the latest trends, user preferences, and contextual information.
Confluent Cloud Networking | Rajan Sundaram, ConfluentHostedbyConfluent
Introduction to networking options available in Confluent Cloud Self Serve provisioning of confluent Kafka clusters. VPC Peering, VNet Peering, Transit Gateway and Private Link Options for AWS, GCP, Azure networking offering. Caveats of confluent's cloud networking solutions customers should be aware of. Details of two major pieces of the architecture of Confluent Cloud - Data Plane Network and Control Plane.
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeksconfluent
As businesses strive to stay at the forefront of innovation, the ability to quickly develop scalable Generative AI (GenAI) applications is essential. Join us for an exclusive webinar featuring MIA Platform, MongoDB, and Confluent, where you'll learn how to compose GenAI apps with real-time data integration in a fraction of the time.
Discover how these three powerful platforms work together to ensure applications remain responsive, relevant, and adaptive to user preferences and contextual changes. Our experts will guide you through leveraging MIA Platform's microservices architecture and low-code development, MongoDB's flexibility, and Confluent's stream processing capabilities. Experience live demonstrations and practical insights that will transform your approach to AI-driven app development, enabling you to accelerate your development process from weeks to mere minutes. Don't miss this opportunity to keep your business at the cutting edge.
Von JavaEE auf Microservice in 6 Monaten - The Good, the Bad, and the wtfs...André Goliath
This document summarizes a talk about transitioning from JavaEE monoliths to microservices architecture in 6 months. It discusses the reasons for moving to microservices (faster development and deployment, lower costs), and the challenges including organizing configuration, communication between services, and deployment. It then outlines the steps taken to implement microservices at a company, including setting up continuous integration, using Spring Boot and Cloud, and establishing vertical feature teams to overcome organizational barriers. The key lessons are that the transition does not require a "big bang", can start with a single service, and works best by automating the development and deployment process from development to production.
Technical Deep Dive: Using Apache Kafka to Optimize Real-Time Analytics in Fi...confluent
Watch this talk here: https://www.confluent.io/online-talks/using-apache-kafka-to-optimize-real-time-analytics-financial-services-iot-applications
When it comes to the fast-paced nature of capital markets and IoT, the ability to analyze data in real time is critical to gaining an edge. It’s not just about the quantity of data you can analyze at once, it’s about the speed, scale, and quality of the data you have at your fingertips.
Modern streaming data technologies like Apache Kafka and the broader Confluent platform can help detect opportunities and threats in real time. They can improve profitability, yield, and performance. Combining Kafka with Panopticon visual analytics provides a powerful foundation for optimizing your operations.
Use cases in capital markets include transaction cost analysis (TCA), risk monitoring, surveillance of trading and trader activity, compliance, and optimizing profitability of electronic trading operations. Use cases in IoT include monitoring manufacturing processes, logistics, and connected vehicle telemetry and geospatial data.
This online talk will include in depth practical demonstrations of how Confluent and Panopticon together support several key applications. You will learn:
-Why Apache Kafka is widely used to improve performance of complex operational systems
-How Confluent and Panopticon open new opportunities to analyze operational data in real time
-How to quickly identify and react immediately to fast-emerging trends, clusters, and anomalies
-How to scale data ingestion and data processing
-Build new analytics dashboards in minutes
Unlock value with Confluent and AWS.pptxAhmed791434
This document discusses building real-time analytics pipelines using Confluent and AWS. It describes how Confluent makes real-time data streams a top priority through features like Apache Kafka. It also explains how to build a typical real-time data pipeline using an event streaming platform to capture, process, and analyze streaming data in real-time. Additionally, it covers how Confluent and AWS can be used together to unlock various use cases across industries through deep integrations and services. Finally, it provides an overview of different stream processing options on AWS including ksqlDB, Kafka Streams, Kinesis Data Analytics, and AWS Lambda.
Confluent Partner Tech Talk with Synthesisconfluent
A discussion on the arduous planning process, and deep dive into the design/architectural decisions.
Learn more about the networking, RBAC strategies, the automation, and the deployment plan.
Building Real-Time Gen AI Applications with SingleStore and Confluentconfluent
Discover how SingleStore and Confluent together create a powerful foundation for real-time generative AI applications. Learn how SingleStore's high-performance data platform and Confluent integrate to process and analyze streaming data in real-time. We'll explore real-world, innovative solutions and show you how SingleStore + Confluent can unlock new gen AI opportunities with your clients.
Cloudfoundry + K8S : Prendre le meilleur des deux mondesErwan Bornier
This document summarizes a presentation about using CloudFoundry and Kubernetes together. It discusses how CloudFoundry BOSH can provide release engineering, deployment, and lifecycle management for Kubernetes clusters through a project called CF Container Runtime (CFCR). This allows using Kubernetes for container orchestration while leveraging BOSH capabilities. It also presents Pivotal's approach to providing a unified platform for developing, deploying and managing applications on CloudFoundry and Kubernetes through shared services, security, networking and other capabilities.
This document discusses the growing importance of APIs and integration in enabling composable and disaggregated architectures. It makes three key points:
1) Consumer demand and supplier strategies are driving increased disaggregation of applications and services, requiring hybrid integration between on-prem and cloud. APIs are the "glue" enabling this integration.
2) APIs are increasingly important both for external access of services but also internal integration within organizations. They represent digital business capabilities that can be marketed and monetized.
3) A cellular architecture modeling organizations and systems as composable, independently deployable "cells" can help overcome integration challenges and enable agility at scale. The WSO2 API Microgateway and Cellery platform aim
This document discusses service oriented cloud computing. It defines cloud computing as dynamically scalable shared resources accessed over a network that users only pay for what they use. The architecture of cloud computing involves communication between clouds. An example is given of an industrial cloud containing a software control system, database storage, application server, and computer network that allows clients to access the site from anywhere in the world. Requirements for implementing a service oriented cloud computing application are provided.
Your Business, as APIs Leveraging Abstractions to Drive the Experience Economy Asanka Abeysinghe
In this session, Asanka will explore how leveraging the correct level of abstractions can drive agile software manufacturing and improve customer experiences in the experience economy. By viewing their business as a set of APIs and using other abstraction layers, businesses can build modular and customer-centric systems that enable rapid innovation and growth. Attendees will gain insights into real-world examples of businesses leveraging APIs to drive growth and competitiveness. They will leave with practical strategies and tactics for streamlining software manufacturing processes and unlocking new revenue streams in the experience economy.
This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservcies Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Migration, backup and restore made easy using Kannikaconfluent
In this presentation, you’ll discover how easily you can migrate data from any Kafka-compatible event hub to Confluent using Kannika’s intuitive self-service interface. We’ll guide you through the process, showing how the same approach can be applied to define specific event data sets and effortlessly spin up secure environments for demos, testing, or other purposes.
You’ll also learn how to back up event data in just a few steps by transferring compressed data to the cloud storage location of your choice. In addition, we’ll demonstrate how to restore filtered datasets of topics, ensuring quick recovery and maintaining business continuity when needed.
Five Things You Need to Know About Data Streaming in 2025confluent
Topics that Peter covers:
Tapping into the Potential of Data Products: Data drives some of today's most important business use cases. Data products enable instant access to reliable and trustworthy data by eliminating the data mess created by point-to-point connections.
The Need to Tap into 'Quick Thinking': The C-level has to reorient itself so it doesn't become the bottleneck to adaptability in a data-driven world. Nine in 10 (90%) business leaders say they must now react in real-time. Learn what you can do to provide executive access to real-time data to enable 'Quick Thinking.'
Rise Above Data Hurdles: Discover how to enforce governance at data production. Reestablishing trustworthiness later is almost always harder, so investing in data tools that solve business problems rather than add to them is essential.
Paradigm to Shift Left: Shift Left is a new paradigm for processing and governing data at any scale, complexity, and latency. Shift Left moves the processing and governance of data closer to the source, enabling organisations to build their data once, build it right and reuse it anywhere within moments of its creation.
The Need for a Strategic View: The positive correlation between data streaming maturity and significant business returns underscores the importance of a long-term, strategic view of data streaming investments. It also highlights the value of advancing beyond initial, siloed use cases to a more integrated approach that leverages data streaming across the enterprise.
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...confluent
In this presentation, we’ll demonstrate how Confluent and Lightstreamer come together to tackle the last-mile challenge of extending your Kafka architecture to web and mobile platforms.
Learn how to effortlessly build real-time web applications within minutes, subscribing to Kafka topics directly from your web pages, with unmatched low latency and high scalability.
Explore how Confluent's leading Kafka platform and Lightstreamer's intelligent proxy work seamlessly to bridge Kafka with the internet frontier, delivering data in real-time.
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent
Confluent per il settore FSI:
- Cos'è il Data Streaming e perché la tua azienda ne ha bisogno
- Chi siamo e come Confluent può aiutarti:
- Rendere Kafka ampiamente accessibile
- Stream, Connect, Process e Governance
- Deep dive sulle soluzioni tecnologiche implementate all'interno della Data Streaming Platform
- Dalla teoria alla pratica: applicazioni reali delle architetture FSI
Data in Motion Tour 2024 Riyadh, Saudi Arabiaconfluent
Data streaming platforms are becoming increasingly important in today’s fast-paced world. From retail giants who need to monitor inventory levels to ensure stores never run out of items, to new-age, innovative banks who are building out-of-the-box banking solutions for traditional retail banks, data streaming platforms are at the centre, powering these workflows.
Data streaming platforms connect all your applications, systems, and teams with a shared view of the most up-to-date, real-time data. From Gen AI, stream governance to stream processing - it’s these cutting edge developments that will be featured during the day.
Unlocking value with event-driven architecture by Confluentconfluent
Sfrutta il potere dello streaming di dati in tempo reale e dei microservizi basati su eventi per il futuro di Sky con Confluent e Kafka®.
In questo tech talk esploreremo le potenzialità di Confluent e Apache Kafka® per rivoluzionare l'architettura aziendale e sbloccare nuove opportunità di business. Ne approfondiremo i concetti chiave, guidandoti nella creazione di applicazioni scalabili, resilienti e fruibili in tempo reale per lo streaming di dati.
Scoprirai come costruire microservizi basati su eventi con Confluent, sfruttando i vantaggi di un'architettura moderna e reattiva.
Il talk presenterà inoltre casi d'uso reali di Confluent e Kafka®, dimostrando come queste tecnologie possano ottimizzare i processi aziendali e generare valore concreto.
Il Data Streaming per un’AI real-time di nuova generazioneconfluent
Per costruire applicazioni di AI affidabili, sicure e governate occorre una base dati in tempo reale altrettanto solida. Ancor più quando ci troviamo a gestire ingenti flussi di dati in continuo movimento.
Come arrivarci? Affidati a una vera piattaforma di data streaming che ti permetta di scalare e creare rapidamente applicazioni di AI in tempo reale partendo da dati affidabili.
Scopri di più! Non perdere il nostro prossimo webinar durante il quale avremo l’occasione di:
• Esplorare il paradigma della GenAI e di come questa nuova tecnnologia sta rimodellando il panorama aziendale, rispondendo alla necessità di offrire un contesto e soluzioni in tempo reale che soddisfino le esigenze della tua azienda.
• Approfondire le incertezze del panorama dell'AI in evoluzione e l'importanza cruciale del data streaming e dell'elaborazione dati.
• Vedere in dettaglio l'architettura in continua evoluzione e il ruolo chiave di Kafka e Confluent nelle applicazioni di AI.
• Analizzare i vantaggi di una piattaforma di streaming dei dati come Confluent nel collegare l'eredità legacy e la GenAI, facilitando lo sviluppo e l’utilizzo di AI predittive e generative.
Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent
Connectors integrate Apache Kafka® with external data systems, enabling you to move away from a brittle spaghetti architecture to one that is more streamlined, secure, and future-proof. However, if your team still spends multiple dev cycles building and managing connectors using just open source Kafka Connect, it’s time to consider a faster and cost-effective alternative.
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
In our exclusive webinar, you'll learn why event-driven architecture is the key to unlocking cost efficiency, operational effectiveness, and profitability. Gain insights on how this approach differs from API-driven methods and why it's essential for your organization's success.
Santander Stream Processing with Apache Flinkconfluent
Flink is becoming the de facto standard for stream processing due to its scalability, performance, fault tolerance, and language flexibility. It supports stream processing, batch processing, and analytics through one unified system. Developers choose Flink for its robust feature set and ability to handle stream processing workloads at large scales efficiently.
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
In today's data-driven world, the Internet of Things (IoT) is revolutionizing industries and unlocking new possibilities. Join Data Reply, Confluent, and Imply as we unveil a comprehensive solution for IoT that harnesses the power of real-time insights.
Workshop híbrido: Stream Processing con Flinkconfluent
El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real.
Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real.
En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...confluent
Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace.
In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms.
You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes.
Don't miss out on this opportunity to learn from industry experts and take your business to the next level.
La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.
Eventos y Microservicios - Santander TechTalkconfluent
Durante esta sesión examinaremos cómo el mundo de los eventos y los microservicios se complementan y mejoran explorando cómo los patrones basados en eventos nos permiten descomponer monolitos de manera escalable, resiliente y desacoplada.
Technology Trends in 2025: AI and Big Data AnalyticsInData Labs
At InData Labs, we have been keeping an ear to the ground, looking out for AI-enabled digital transformation trends coming our way in 2025. Our report will provide a look into the technology landscape of the future, including:
-Artificial Intelligence Market Overview
-Strategies for AI Adoption in 2025
-Anticipated drivers of AI adoption and transformative technologies
-Benefits of AI and Big data for your business
-Tips on how to prepare your business for innovation
-AI and data privacy: Strategies for securing data privacy in AI models, etc.
Download your free copy nowand implement the key findings to improve your business.
DevOpsDays Atlanta 2025 - Building 10x Development Organizations.pptxJustin Reock
Building 10x Organizations with Modern Productivity Metrics
10x developers may be a myth, but 10x organizations are very real, as proven by the influential study performed in the 1980s, ‘The Coding War Games.’
Right now, here in early 2025, we seem to be experiencing YAPP (Yet Another Productivity Philosophy), and that philosophy is converging on developer experience. It seems that with every new method we invent for the delivery of products, whether physical or virtual, we reinvent productivity philosophies to go alongside them.
But which of these approaches actually work? DORA? SPACE? DevEx? What should we invest in and create urgency behind today, so that we don’t find ourselves having the same discussion again in a decade?
Mobile App Development Company in Saudi ArabiaSteve Jonas
EmizenTech is a globally recognized software development company, proudly serving businesses since 2013. With over 11+ years of industry experience and a team of 200+ skilled professionals, we have successfully delivered 1200+ projects across various sectors. As a leading Mobile App Development Company In Saudi Arabia we offer end-to-end solutions for iOS, Android, and cross-platform applications. Our apps are known for their user-friendly interfaces, scalability, high performance, and strong security features. We tailor each mobile application to meet the unique needs of different industries, ensuring a seamless user experience. EmizenTech is committed to turning your vision into a powerful digital product that drives growth, innovation, and long-term success in the competitive mobile landscape of Saudi Arabia.
Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results.
News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?
Quantum Computing Quick Research Guide by Arthur MorganArthur Morgan
This is a Quick Research Guide (QRG).
QRGs include the following:
- A brief, high-level overview of the QRG topic.
- A milestone timeline for the QRG topic.
- Links to various free online resource materials to provide a deeper dive into the QRG topic.
- Conclusion and a recommendation for at least two books available in the SJPL system on the QRG topic.
QRGs planned for the series:
- Artificial Intelligence QRG
- Quantum Computing QRG
- Big Data Analytics QRG
- Spacecraft Guidance, Navigation & Control QRG (coming 2026)
- UK Home Computing & The Birth of ARM QRG (coming 2027)
Any questions or comments?
- Please contact Arthur Morgan at [email protected].
100% human made.
TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc
Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency.
This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data.
Attendees will learn:
- Consumer awareness around data brokers and what consumers are doing to limit data collection
- How businesses assess third-party vendors and their consent management operations
- Where business preparedness needs improvement
- What these trends mean for the future of privacy governance and public trust
This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.
HCL Nomad Web – Best Practices and Managing Multiuser Environmentspanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-and-managing-multiuser-environments/
HCL Nomad Web is heralded as the next generation of the HCL Notes client, offering numerous advantages such as eliminating the need for packaging, distribution, and installation. Nomad Web client upgrades will be installed “automatically” in the background. This significantly reduces the administrative footprint compared to traditional HCL Notes clients. However, troubleshooting issues in Nomad Web present unique challenges compared to the Notes client.
Join Christoph and Marc as they demonstrate how to simplify the troubleshooting process in HCL Nomad Web, ensuring a smoother and more efficient user experience.
In this webinar, we will explore effective strategies for diagnosing and resolving common problems in HCL Nomad Web, including
- Accessing the console
- Locating and interpreting log files
- Accessing the data folder within the browser’s cache (using OPFS)
- Understand the difference between single- and multi-user scenarios
- Utilizing Client Clocking
Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark?
At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍
Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPathCommunity
Join this UiPath Community Berlin meetup to explore the Orchestrator API, Swagger interface, and the Test Manager API. Learn how to leverage these tools to streamline automation, enhance testing, and integrate more efficiently with UiPath. Perfect for developers, testers, and automation enthusiasts!
📕 Agenda
Welcome & Introductions
Orchestrator API Overview
Exploring the Swagger Interface
Test Manager API Highlights
Streamlining Automation & Testing with APIs (Demo)
Q&A and Open Discussion
Perfect for developers, testers, and automation enthusiasts!
👉 Join our UiPath Community Berlin chapter: https://community.uipath.com/berlin/
This session streamed live on April 29, 2025, 18:00 CET.
Check out all our upcoming UiPath Community sessions at https://community.uipath.com/events/.
Dev Dives: Automate and orchestrate your processes with UiPath MaestroUiPathCommunity
This session is designed to equip developers with the skills needed to build mission-critical, end-to-end processes that seamlessly orchestrate agents, people, and robots.
📕 Here's what you can expect:
- Modeling: Build end-to-end processes using BPMN.
- Implementing: Integrate agentic tasks, RPA, APIs, and advanced decisioning into processes.
- Operating: Control process instances with rewind, replay, pause, and stop functions.
- Monitoring: Use dashboards and embedded analytics for real-time insights into process instances.
This webinar is a must-attend for developers looking to enhance their agentic automation skills and orchestrate robust, mission-critical processes.
👨🏫 Speaker:
Andrei Vintila, Principal Product Manager @UiPath
This session streamed live on April 29, 2025, 16:00 CET.
Check out all our upcoming Dev Dives sessions at https://community.uipath.com/dev-dives-automation-developer-2025/.
How Can I use the AI Hype in my Business Context?Daniel Lehner
𝙄𝙨 𝘼𝙄 𝙟𝙪𝙨𝙩 𝙝𝙮𝙥𝙚? 𝙊𝙧 𝙞𝙨 𝙞𝙩 𝙩𝙝𝙚 𝙜𝙖𝙢𝙚 𝙘𝙝𝙖𝙣𝙜𝙚𝙧 𝙮𝙤𝙪𝙧 𝙗𝙪𝙨𝙞𝙣𝙚𝙨𝙨 𝙣𝙚𝙚𝙙𝙨?
Everyone’s talking about AI but is anyone really using it to create real value?
Most companies want to leverage AI. Few know 𝗵𝗼𝘄.
✅ What exactly should you ask to find real AI opportunities?
✅ Which AI techniques actually fit your business?
✅ Is your data even ready for AI?
If you’re not sure, you’re not alone. This is a condensed version of the slides I presented at a Linkedin webinar for Tecnovy on 28.04.2025.
AI and Data Privacy in 2025: Global TrendsInData Labs
In this infographic, we explore how businesses can implement effective governance frameworks to address AI data privacy. Understanding it is crucial for developing effective strategies that ensure compliance, safeguard customer trust, and leverage AI responsibly. Equip yourself with insights that can drive informed decision-making and position your organization for success in the future of data privacy.
This infographic contains:
-AI and data privacy: Key findings
-Statistics on AI data privacy in the today’s world
-Tips on how to overcome data privacy challenges
-Benefits of AI data security investments.
Keep up-to-date on how AI is reshaping privacy standards and what this entails for both individuals and organizations.
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxshyamraj55
We’re bringing the TDX energy to our community with 2 power-packed sessions:
🛠️ Workshop: MuleSoft for Agentforce
Explore the new version of our hands-on workshop featuring the latest Topic Center and API Catalog updates.
📄 Talk: Power Up Document Processing
Dive into smart automation with MuleSoft IDP, NLP, and Einstein AI for intelligent document workflows.
Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies
In late April 2025, a significant portion of Europe, particularly Spain, Portugal, and parts of southern France, experienced widespread, rolling power outages that continue to affect millions of residents, businesses, and infrastructure systems.
Increasing Retail Store Efficiency How can Planograms Save Time and Money.pptxAnoop Ashok
In today's fast-paced retail environment, efficiency is key. Every minute counts, and every penny matters. One tool that can significantly boost your store's efficiency is a well-executed planogram. These visual merchandising blueprints not only enhance store layouts but also save time and money in the process.
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company
Explore the benefits and features of advanced logistics management software for businesses in Riyadh. This guide delves into the latest technologies, from real-time tracking and route optimization to warehouse management and inventory control, helping businesses streamline their logistics operations and reduce costs. Learn how implementing the right software solution can enhance efficiency, improve customer satisfaction, and provide a competitive edge in the growing logistics sector of Riyadh.
Complete Guide to Advanced Logistics Management Software in Riyadh.pdfSoftware Company
스타트업을 위한 Confluent 세미나
1. Part 1. Kafka, 어떻게 시작할까?
Confluent Startup Webinar Series
간종석 - Account Executive, Confluent
최영일 - Data Engineer, Bagelcode
Engin Cukuroglu, PhD - Solution Engineer, Confluent
황주필 - Solution Engineer, Confluent
2. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Part 1. Why Kafka for Digital Native Business?
Part 2. 고객과의 대화 - Bagelcode’s Success Story
Part 3. Scenario 기반 Demo
Q&A
Today’s Contents
4. The Rise of Event Streaming
2010
Apache Kafka
Confluent 설립자들이
LinkedIn에서 개발
2014
2020
70%
Fortune 500
기업이 신뢰하고
사용중인
Apache Kafka
4
5. The Rise of Data in Motion
금융 & 은행 보험 통신
여행 & 리테일
10 OUT
OF 10 8 OUT
OF 8
Fortune 500 Companies
Using Apache Kafka
70%
운송수단 에너지 & 유틸리티 엔터테인먼트
기술
8 OUT
OF 10 9 OUT
OF 10
10 OUT
OF 10
10 OUT
OF 10
10 OUT
OF 10 8 OUT
OF 10
6. Real-time
Data
구매/세일즈
배송
트레이드
고객 경험
Data in Motion: 새로운 패러다임
Continuously Process Streams of Data in Real-time
“우리는 데이터 뿐만아니라 우리의 생각을 at rest 하는
방식에서 in motion의 형태로 전환해야 했습니다.” —
실시간 스트림 프로세싱
풍부한 프론트엔드 고객 경험
실시간 백앤드 운영
7. Modernize
your apps
차세대 아키텍처가
지원하는 Realtime
Insight로
애플리케이션의 가치를
높이십시오
데이터 통합
Database changes
Log
events
IoT
events
Web events
Connected car
Fraud detection
Customer 360
Personalized
promotions
실시간 데이터로
구동되는 앱
Quality
assurance
SIEM/SOC
Inventory
management
Proactive
patient care
Sentiment
analysis
Capital
management
Modernize
Your Apps
Amazon
Kinesis
Amazon
S3
7
9. Why Apache Kafka for Digital Native Business?
Time to Market
가속화
Confluent로 시작한 후 1 주 이내에
대규모 Kafka 배포
TCO 절감 효과
(Total Cost Of Ownership)
낮은 인프라 비용, 유지 보수 및 다운
타임 위험을 줄이고, 보다 효율적으로
운영
ROI 극대화
(Return On Investment)
프로젝트를 더 빠르게 시작하고
실시간 데이터를 활용하여 더 높은
수익을 제공
9
10. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Cloud-native: Confluent Cloud completes Apache
Kafka as a Cloud native Kafka Service
무제한 데이터 저장 용량
제공으로 실시간 앱 및
유스케이스에
더 큰 데이터 세트를 제공
즉시 확장(scale-up)하고
인프라 오버 프로비저닝을
방지하기 위해 축소(scale-
down)
다양한 글로벌 인프라
환경에서 클러스터를
운영하여 조직 내에
일관된 데이터 페브릭
제공
탄력적인 확장 글로벌 인프라 무제한 용량
단순한 kafka를 넘어서
완전한 Data in Motion
플랫폼으로 더 빠르게
구축 및 운영
완전
관리형
11. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Part 2. 고객과의 대화 - Bagelcode
최영일
Tech Lead - Data Engineering,
Bagelcode
간종석
Account Executive for Startups,
Confluent
12. Part 3. Scenario 기반 Demo
Engin Cukuroglu, PhD - Solution Engineer, Confluent
황주필 - Solution Engineer, Confluent
13. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Monoliths vs Microservices
Microservices (마이크로서비스)
Independently deployable and upgradable (독립적인 배포 및
업그레이드)
Scalable and agile = faster feature delivery (빠른 기능 구현 가능)
Leverages new technologies (새로운 기술 적용 용이)
Monoliths (모놀리틱)
❖ Scalability issues (스케일 확장이 어려움)
❖ Difficult to change (아키택쳐 변경의 어려움)
❖ Difficult to adopt new technologies (새로운 기술 추가에
제약)
14. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
How to create Problems? (아키택쳐 디자인 중간에 발생하는
문제들)
Ap
ps
D
W
Ap
ps
Ap
ps
D
B
• I know how to create an app
(어떻게 앱을 만드는지 알고 있습니다.)
• I need database and monitoring
(데이터베이스 구성과 모니터링이 필요합니다.)
• Look! I created another app and I need
DataWarehouse
(새로운 앱을 생성했고, 데이터웨어하우스가 필요합니다!)
• Here goes another app
(다른 앱을 또 만들었어요!)
• I heard microservices are cool!
(요즘에 마이크로서비스가 좋다던데요?!)
15. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
How to create Problems?
15
16. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
How to confuse the engineers?
D
W
Ap
ps
Ap
ps
D
B
Sa
as
Sa
as
Sa
as
Sa
as
Sa
as
Sa
as
D
W
D
B
D
B
D
B
D
B
Ap
ps
Ap
ps
Ap
ps
Ap
ps
O
n
Prem O
n
Cloud
17. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
How do engineers solve this problem?
17
18. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
How do the engineers solve this problem?
18
19. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
How do the engineers solve this problem?
DW
Apps
Apps
DB
Saas Saas
DW
D
B
Apps
Apps
21. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
21
22. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
22
Brokers
23. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
23
24. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
24
Consumer Producer Brokers
25. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
25
26. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
26
Consumer Producer
Kafka Streams
Brokers
27. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
27
28. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
28
Consumer Produce
r
Kafka Streams
KsqlDB
Brokers
29. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
29
30. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
30
Consumer Producer
Kafka Streams
KsqlDB Schema
Registry
Brokers
31. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
31
32. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
32
Consumer Producer
Kafka Streams
KsqlDB Schema
Registry
Kafka
Connect
Brokers
33. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
33
34. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Kafka Architecture
34
Consumer Producer
Kafka Streams
KsqlDB Schema
Registry
Kafka
Connect
Brokers
Rest Proxy
36. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
Bookshop Order Management (서점 주문 관리)
36
Bookshop Order Management
Visualize
(시각화)
Create
Order
(주문 생성)
Order
Details
(주문 정보)
Shipping
Details
(배송 정보)
37. Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc. 37
Schema
Registry
(스키마 레지스트리)
Brokers
(카프카
브로커)
Admin
(관리자)
Register Schemas
(스키마 등록)
Create Topics
(토픽 생성)
ShipBooks-
consumer
(배송 컨슈머)
CachedSR
Consume
ShipBooks-producer
(배송 프로듀서)
Produce
Analyze
(분석)
Visualize
(시각화)
BookOrder-producer
(주문 프로듀서)
CachedSR
Produce
Retrieve Schemas
(스키마 검색)
Kafka
Connect
(카프카 커넥트)
Consume
Retrieve Schemas
(스키마 검색)
Bookshop Order Management (서점 주문 관리)
39. 지금 바로 시작하세요 !
Confluent Cloud 계정
생성 - $400 크레딧 제공
https://www.confluent.io/get-
started/
1:1 솔루션 디자인
Office Hour 진행 설문지 피드백