Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs | Sheikh Araf and Ameya Panse, Goldman Sachs

Jun 8, 2021Download as PPTX, PDF0 likes1,543 views

In our payments platform at Goldman Sachs Transaction Banking, Apache Kafka plays a critical role as the messaging bus in our micro-services architecture. Being a part of the financial service industry we need to ensure high-availability of our platform and quick response time during failures. In this talk we will explore how we monitor and alert on the health of our Kafka clusters using our heartbeat application and clients using DataDog dashboards. We will see how we consolidate JMX metrics such as error-rates, connection-rates, latencies and consumer lag from all producers and consumers using JMX agent sidecar to provide a live view of the health of our entire infrastructure. We will also discuss our culture of game days where we regularly test the resiliency of all the clients in our infrastructure by simulating various failure scenarios to improve the overall availability of our infrastructure.

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Monitoring and Resiliency
Testing our Apache Kafka
Clusters
Ameya Panse
Associate
Goldman Sachs
Sheikh Araf
Associate
Goldman Sachs

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
We’re helping clients build a treasury of the future and powering
software partners to enhance their offerings.
Differentiated Platform

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Apache Kafka Backbone
Apache Kafka Cluster
Payment
Service
Ledger Reporting
Validation
Data
Platform
Payment
Rails
CRM

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Two Sides to Monitoring Kafka Infrastructure
Kafka
Cluster

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
App App App
Kafka
Cluster
DB
DB
App App App
App
App
Connectors
Stream
Processors
Consumers
Producers
Two Sides to Monitoring Kafka Infrastructure

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Kafka Brokers
• CPU Usage
• Disk Usage
• Network Tx Packets
• Network Rx Packets
• Leader Count
• Under Replicated Partition
Count
Kafka Clients
• Producer Error Rate
• Active Consumer Connections
• Producer Retry Rate
• Consumer Connection Close
Rate
• Produce & Consume Latency
Two Sides to Monitoring Kafka Infrastructure

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Monitoring the Clients

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Collecting Client Metrics
Containers
Application
Container
JMX Agent
Sidecar
Virtual Machine
Application
Process
JMX Agent
Process
Datadog
Backend
Datadog
Dashboard
PagerDuty
Alerts

$arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx Frictionless onboarding module "my-app" { source = "..." ... kafka_app_name = "my-service" } main.tf Behind the scenes: • Add a JMX agent sidecar • Configure agent to collect Kafka client metrics • Configure agent to authenticate and push metrics to dashboard$

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Monitoring TLS Certificate Expiry
<dependency>
<groupId>com.gs.txb.security</groupId>
<artifactId>certificate-info-endpoint</artifactId>
<version>1.0.0</version>
</dependency>
pom.xml
management.server.port=PORT
management.server.keystore=KeystoreLocation
application.properties
Containers
Application Container
JMX Agent Sidecar
/certs
Poll
Metric

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Monitoring the Cluster

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Synthetic Monitoring with Heartbeat App
• Broker Availability
• Cross-Zone Connectivity
• Monitor Support Services
• Real Time Alerting
• End to End Monitoring

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Synthetic Monitoring Algorithm
Kafka Cluster
Broker 1
Broker 2
Broker 3
Partition 1
Leader
Partition 2
Leader
Partition 3
Leader
Producer 1
JMX
Sidecar
Producer 2
JMX
Sidecar
Producer 3
JMX
Sidecar
Producer Containers
Data Center 1
Data Center 2
Data Center 3
Consumer 1
JMX
Sidecar
Consumer 2
JMX
Sidecar
Consumer 3
JMX
Sidecar
Consumer Containers
Dashboard

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Consolidating the Metrics
Dashboards and Alerts

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Alerts

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Putting it all together
Culture of Resiliency Game Days

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Failure Scenarios
Zone 1
Kafka Cluster
Broker Broker
Zone 2
Broker Broker
Zone 3
Broker Broker
Zone 4
Broker Broker
Zone 1
Kafka Cluster
Broker Broker
Zone 2
Broker Broker
Zone 3
Broker Broker
Zone 4
Broker Broker
Zone 1
Kafka Cluster
Broker Broker
Zone 2
Broker Broker
Zone 3
Broker Broker
Zone 4
Broker Broker
Loss of one broker in one zone Loss of two broker in one zone Loss of two broker in different zones
Topic: Replication factor = 3
Cluster: Min in-sync replicas = 2
Producer: Acks = all

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Game Days
Business Services
Transaction Banking Stack
Apache Kafka Backbone
App App App App
Stream of
Payment Messages
Track system health
via Kafka dashboard
Assert all payments
are successful
Ensure all
applications recover
automatically

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
In Summary…
• Apache Kafka as backbone for processing payments
• Monitor cluster health with synthetic traffic
• Monitoring clients using JMX agent sidecar
• Simplify the onboarding process to improve monitoring coverage
• One stop view to monitor the health of the infrastructure
• A culture of regular resiliency testing

arafshhomeMy DocumentsMonitoring and Resiliency Testing our Apache Kafka Clusters Send 1.pptx
Thank You
Ameya Panse
ameya.panse@gs.com
Sheikh Araf
sheikh.araf@gs.com

Google Cloud offers several fully managed database services for PostgreSQL workloads, including Cloud SQL and AlloyDB. Cloud SQL provides a fully managed relational database service for PostgreSQL, MySQL, and SQL Server. It offers 99.999% availability, unlimited scaling, and automatic failure recovery. AlloyDB is a new database engine compatible with PostgreSQL that provides up to 4x faster transactions and 100x faster analytics queries than standard PostgreSQL. It features independent scaling of storage and computing resources. Google Cloud aims to be the best home for PostgreSQL workloads by providing compatibility with open source PostgreSQL and enterprise-grade features, performance, reliability, and support across its database services.

Data Con LA 2022 - Pre- Recorded - OpenSearch: Everything You Need to Know Ab...Data Con LA

Seth Muthukaruppan, Consultant at Instacluster Data Engineering OpenSearch is an incredibly powerful search engine and analytics suite for ingesting, searching, visualizing, and analyzing your data and it is fully open source. This Apache 2.0-licensed and community-driven collection of technologies harnesses an architecture that combines the powers of Elasticsearch 7.10.2, Kibana 7.10.2 and Apache Lucene. With OpenSearch, users gain a distributed framework featuring particularly powerful scalability, high availability, and database-like capabilities. Attendees at this DataCon LA presentation will come away understanding OpenSearch's architecture and its building-block technology components, including: -- Apache Lucene utilization. Learn how this high-performance Java-based search library utilizes Lucene's inverted search index to delivers incredibly fast search results (while supporting natural language, wildcard, fuzzy, and proximity searches). -- OpenSearch cluster architecture. An OpenSearch cluster is a distributed and horizontally-scalable collection of nodes, which are differentiated based on the operations they perform. Attendees will learn the specific functions of master, master-eligible, data, client, ingest nodes. -- Data organization. Understand how OpenSearch organizes data into indices (which contain documents, which contain fields). -- Internal data structures. Get an in-depth look at how OpenSearch achieves scalability and reliability by breaking up indices into shards and segments, and utilizes translogs. -- Aggregations. See how OpenSearch enables its advanced built-in analytics capabilities through the power of aggregations.

AWS Route53zekeLabs Technologies

This document provides an overview of Amazon Route 53 DNS services including: - IPv4 and IPv6 address spaces and how Route 53 resolves domain names to IP addresses using A records. - Common DNS record types like NS, SOA, CNAME and how they work. - Route 53 routing policies for controlling traffic like simple, weighted, latency, failover and geolocation routing. - How alias records can simplify configuration by automatically reflecting changes to referenced resources. - A example of setting up Route 53 with domains, record sets, Elastic Load Balancers and instances across regions.

OSMC 2021 | Introduction into OpenSearchNETWAYS

OpenSearch is a community-driven, open source search and analytics suite derived from Elasticsearch 7.10.2 and Kibana 7.10.2. It was initiated by AWS and consists of the OpenSearch search engine and OpenSearch Dashboards visualization interface. OpenSearch aims to provide a true open source search and analytics engine following licensing changes by Elastic that removed Elasticsearch's open source status.

MongoDB WiredTiger InternalsNorberto Leite

Real-Life Use Cases & Architectures for Event Streaming with Apache KafkaKai Wähner

Streaming all over the World: Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka. Learn about various case studies for event streaming with Apache Kafka across industries. The talk explores architectures for real-world deployments from Audi, BMW, Disney, Generali, Paypal, Tesla, Unity, Walmart, William Hill, and more. Use cases include fraud detection, mainframe offloading, predictive maintenance, cybersecurity, edge computing, track&trace, live betting, and much more.

Elasticsearch in NetflixDanny Yuan

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop. It's also enabling many real-time system frameworks and use cases. Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API. Also talk about the best practices involved in running a producer/consumer. In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects. We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing Kafka ACLs and monitoring Consumer offsets.

쿠키런: 킹덤 대규모 인프라 및 서버 운영 사례 공유 [데브시스터즈 - 레벨 200] - 발표자: 용찬호, R&D 엔지니어, 데브시스터즈 ...Amazon Web Services Korea

<쿠키런:킹덤> 게임 유저 수가 급증하면서 지금까지 겪어보지 못했던 대규모 인프라 환경을 운영하게 되었고, 그 과정에서 다양한 문제점들에 부딪히게 되었습니다. 이 세션에서는 AWS에서 Stateful 한 게임 서버를 어떻게 운영해야 하는지 아키텍처 관점에서 먼저 설명한 후, 수 백만 명의 사용자를 감당하기 위해 해결해야 했던 어려움에 대해 Scalability 관점에서 설명해드립니다.

Kappa vs Lambda Architectures and Technology ComparisonKai Wähner

Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers. This video explores why a single real-time pipeline, called Kappa architecture, is the better fit for many enterprise architectures. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for a Lambda architecture. The main focus of the discussion is on Apache Kafka (and its ecosystem) as the de facto standard for event streaming to process data in motion (the key concept of Kappa), but the video also compares various technologies and vendors such as Confluent, Cloudera, IBM Red Hat, Apache Flink, Apache Pulsar, AWS Kinesis, Amazon MSK, Azure Event Hubs, Google Pub Sub, and more. Video recording of this presentation: https://youtu.be/j7D29eyysDw Further reading: https://www.kai-waehner.de/blog/2021/09/23/real-time-kappa-architecture-mainstream-replacing-batch-lambda/ https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/ https://www.kai-waehner.de/blog/2021/05/09/kafka-api-de-facto-standard-event-streaming-like-amazon-s3-object-storage/

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoophuguk

At Google Cloud Platform, we're combining the Apache Spark and Hadoop ecosystem with our software and hardware innovations. We want to make these awesome tools easier, faster, and more cost-effective, from 3 to 30,000 cores. This presentation will showcase how Google Cloud Platform is innovating with the goal of bringing the Hadoop ecosystem to everyone. Bio: "I love data because it surrounds us - everything is data. I also love open source software, because it shows what is possible when people come together to solve common problems with technology. While they are awesome on their own, I am passionate about combining the power of open source software with the potential unlimited uses of data. That's why I joined Google. I am a product manager for Google Cloud Platform and manage Cloud Dataproc and Apache Beam (incubating). I've previously spent time hanging out at Disney and Amazon. Beyond Google, love data, amateur radio, Disneyland, photography, running and Legos."

What you need to know about Generative AI and Data Management?Denodo

Watch full webinar here: https://buff.ly/3UXy0A2 It should be no surprise that Generative AI will have a profound impact to data management in years to come. Much like other areas of the technology sector, the opportunities presented by GenAI will accelerate our efforts around all aspects of data management, including self-service, automation, data governance and security. On the other hand, it is also becoming clearer that to unleash the true potential of AI assistants powered by GenAI, we need novel implementation strategies and a reimagined data architecture. This presents an exhilarating yet challenging future, demanding innovative thinking and methodologies in data management. Join us on this webinar to learn about: - The opportunities and challenges presented by GenAI today. - Exploiting GenAI to democratize data management. - How to augment GenAI applications with corporate data and knowledge. - How to get started.

Introduction to elasticsearchhypto

Aws route 53Rafael Salerno de Oliveira

DNS converts human-friendly domain names like www.google.com to IP addresses like 8.8.8.8. There are two main IP address formats: IPv4 with 32-bit addresses and IPv4 with 128-bit addresses supporting over 340 undecillion addresses. Top-level domains are controlled by IANA and stored in a root zone database listing available top-level domains. DNS records like SOA, NS, TTL, A, AAAA, CNAME etc. are used to direct traffic and cache records. AWS Route 53 offers routing policies including simple, weighted, latency-based, failover, geolocation and multivalue answer routing.

Building Event Driven (Micro)services with Apache KafkaGuido Schmutz

What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Prometheus 101Paul Podolny

Prometheus is an open-source monitoring system that collects metrics from instrumented systems and applications and allows for querying and alerting on metrics over time. It is designed to be simple to operate, scalable, and provides a powerful query language and multidimensional data model. Key features include no external dependencies, metrics collection by scraping endpoints, time-series storage, and alerting handled by the AlertManager with support for various integrations.

Data pipelines from zero to solidLars Albertsson

This presentation is an attempt do demystify the practice of building reliable data processing pipelines. We go through the necessary pieces needed to build a stable processing platform: data ingestion, processing engines, workflow management, schemas, and pipeline development processes. The presentation also includes component choice considerations and recommendations, as well as best practices and pitfalls to avoid, most learnt through expensive mistakes.

Grafana Loki: like Prometheus, but for LogsMarco Pracucci

Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream. In this talk, we will introduce Loki, its architecture and the design trade-offs in an approachable way. We’ll both cover Loki and Promtail, the agent used to scrape local logs to push to Loki, including the Prometheus-style service discovery used to dynamically discover logs and attach metadata from applications running in a Kubernetes cluster. Finally, we’ll show how to query logs with Grafana using LogQL - the Loki query language - and the latest Grafana features to easily build dashboards mixing metrics and logs.

The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner

This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes: 1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data. 2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels. 3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms. 4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality. 5) Real-time cybersecurity applications that use streaming data

Kafka Quotas Talk at LinkedInAditya Auradkar

The document discusses implementing client quotas in Apache Kafka to ensure fair sharing of broker resources among clients. It describes enforcing quotas based on client IDs and overriding quotas for specific clients. Metrics are used to monitor client throughput and enforce quotas by delaying requests if clients exceed their limits. The approach aims to improve performance for well-behaved clients by preventing a few misbehaving clients from impacting others. Evaluation results show improved throughput and latency for clients when quotas are enforced.

AWS Route53 FundamentalsPiyush Agrawal

Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Edureka!

( ELK Stack Training - https://www.edureka.co/elk-stack-trai... ) This Edureka Elasticsearch Tutorial will help you in understanding the fundamentals of Elasticsearch along with its practical usage and help you in building a strong foundation in ELK Stack. This video helps you to learn following topics: 1. What Is Elasticsearch? 2. Why Elasticsearch? 3. Elasticsearch Advantages 4. Elasticsearch Installation 5. API Conventions 6. Elasticsearch Query DSL 7. Mapping 8. Analysis 9 Modules

Edge architecture ieee international conference on cloud engineeringMikey Cohen - Hiring Amazing Engineers

This document summarizes Netflix's global cloud edge architecture. Key points include: - Netflix uses edge services and a global cloud infrastructure to deliver content to over 1000 device types in over 40 countries. - Zuul is an open source framework that Netflix uses for dynamic routing, authentication, testing, and security across its edge services. - Netflix's edge scripting tier allows device teams to rapidly deploy scripts that control endpoints, content formatting, and APIs for different devices. - RxJava and Hystrix help make the edge service API asynchronous, fault tolerant, and able to handle high concurrency. - Netflix's delivery pipeline uses techniques like canary analysis, debugging, and load testing to continuously and automatically deploy changes

아름답고 유연한 데이터 파이프라인 구축을 위한 Amazon Managed Workflow for Apache Airflow - 유다니엘 A...Amazon Web Services Korea

Apache Airflow는 복잡한 데이터 처리 파이프라인의 전체적인 프로세스를 자동화하기 위한 워크플로우 관리 플랫폼이며 오픈 소스 커뮤니티에서 활발하게 기여하고 있는 top-level 프로젝트 입니다. AWS는 최근에 Amazon Managed Workflow for Apache Airflow (MWAA) 서비스를 정식 출시하였고, 본 강연에서는 Apache Airflow 및 MWAA를 소개하고 어떻게 AWS 서비스와 연동하여 데이터 처리 워크플로우를 구축할 수 있는지 데모를 통해 알려 드립니다.

AWS LambdaScott Leberknight

OSA Con 2022 - Arrow in Flight_ New Developments in Data Connectivity - David...Altinity Ltd

The document discusses the history and development of Apache Arrow, an open-source cross-language development platform for in-memory data. Some key points: - Arrow started in 2016 to optimize data transfer between systems using a standardized columnar memory format. - It has since expanded to include libraries in many languages, file formats like Arrow and Parquet, and distributed computing capabilities like Arrow Flight for RPC. - Over time, more projects have adopted Arrow as an internal data structure for improved performance, including Spark, DuckDB, and Streamlit. - Today Arrow is an ecosystem of interoperable components, with continued work on higher-level tools around databases, machine learning, and geospatial data.

Hadoop REST API Security with Apache Knox GatewayDataWorks Summit

The document discusses the Apache Knox Gateway, which is an extensible reverse proxy framework that securely exposes REST APIs and HTTP-based services from Hadoop clusters. It provides features such as support for common Hadoop services, integration with enterprise authentication systems, centralized auditing of REST API access, and service-level authorization controls. The Knox Gateway aims to simplify access to Hadoop services, enhance security by protecting network details and supporting partial SSL, and enable centralized management and control over REST API access.

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Red Hat Developers

This document discusses Apache Kafka and Red Hat OpenShift Streams for Apache Kafka. It begins with an overview of what Apache Kafka is and its common use cases. It then demonstrates how Red Hat OpenShift Streams provides a managed Apache Kafka cluster as a service, including a dedicated cluster, configuration management, metrics, monitoring and other features to provide a streamlined developer experience. It concludes with information on trying OpenShift Streams for Apache Kafka and additional resources.

Apache kafka-a distributed streaming platformconfluent

More Related Content

What's hot (20)

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

쿠키런: 킹덤 대규모 인프라 및 서버 운영 사례 공유 [데브시스터즈 - 레벨 200] - 발표자: 용찬호, R&D 엔지니어, 데브시스터즈 ...Amazon Web Services Korea

Kappa vs Lambda Architectures and Technology ComparisonKai Wähner

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoophuguk

What you need to know about Generative AI and Data Management?Denodo

Introduction to elasticsearchhypto

Aws route 53Rafael Salerno de Oliveira

Building Event Driven (Micro)services with Apache KafkaGuido Schmutz

Prometheus 101Paul Podolny

Data pipelines from zero to solidLars Albertsson

Grafana Loki: like Prometheus, but for LogsMarco Pracucci

The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner

Kafka Quotas Talk at LinkedInAditya Auradkar

AWS Route53 FundamentalsPiyush Agrawal

Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Edureka!

Edge architecture ieee international conference on cloud engineeringMikey Cohen - Hiring Amazing Engineers

아름답고 유연한 데이터 파이프라인 구축을 위한 Amazon Managed Workflow for Apache Airflow - 유다니엘 A...Amazon Web Services Korea

AWS LambdaScott Leberknight

OSA Con 2022 - Arrow in Flight_ New Developments in Data Connectivity - David...Altinity Ltd

Hadoop REST API Security with Apache Knox GatewayDataWorks Summit

Apache Kafka Best PracticesDataWorks Summit/Hadoop Summit

쿠키런: 킹덤 대규모 인프라 및 서버 운영 사례 공유 [데브시스터즈 - 레벨 200] - 발표자: 용찬호, R&D 엔지니어, 데브시스터즈 ...Amazon Web Services Korea

Kappa vs Lambda Architectures and Technology ComparisonKai Wähner

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoophuguk

What you need to know about Generative AI and Data Management?Denodo

Introduction to elasticsearchhypto

Aws route 53Rafael Salerno de Oliveira

Building Event Driven (Micro)services with Apache KafkaGuido Schmutz

Prometheus 101Paul Podolny

Data pipelines from zero to solidLars Albertsson

Grafana Loki: like Prometheus, but for LogsMarco Pracucci

The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner

Kafka Quotas Talk at LinkedInAditya Auradkar

AWS Route53 FundamentalsPiyush Agrawal

Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Edureka!

Edge architecture ieee international conference on cloud engineeringMikey Cohen - Hiring Amazing Engineers

아름답고 유연한 데이터 파이프라인 구축을 위한 Amazon Managed Workflow for Apache Airflow - 유다니엘 A...Amazon Web Services Korea

AWS LambdaScott Leberknight

OSA Con 2022 - Arrow in Flight_ New Developments in Data Connectivity - David...Altinity Ltd

Hadoop REST API Security with Apache Knox GatewayDataWorks Summit

Similar to Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs | Sheikh Araf and Ameya Panse, Goldman Sachs (20)

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...Red Hat Developers

Apache kafka-a distributed streaming platformconfluent

Apache Kafka - A Distributed Streaming PlatformPaolo Castagna

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...HostedbyConfluent

Event-driven application architectures are becoming increasingly common as a large number of users demand more interactive, real-time, and intelligent responses. Yet it can be challenging to decide how to capture and perform real-time data analysis and deliver differentiating experiences. Join experts from Confluent and AWS to learn how to build Apache Kafka®-based streaming applications backed by machine learning models. Adopting the recommendations will help you establish repeatable patterns for high performing event-based apps.

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Kai Wähner

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

Presentation @ Oracle Code Berlin. Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can we make sure that all these events are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amounts of messages between a source and a target. This session will start with an introduction of Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table.

Kafka Connect & Kafka Streams/KSQL - the ecosystem around KafkaGuido Schmutz

After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL. Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go. Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...Trivadis

IMC Summit 2016 Breakout - Roman Shtykh - Apache Ignite as a Data Processing HubIn-Memory Computing Summit

Today, many companies are faced with a huge quantity of data and a wide variety of tools with which to process it. This potentially allows for great opportunities to satisfy customers’ needs and bring user experience to the next level. However, in order to achieve this and provide a competitive solution, sophisticated and complex data processing is needed. Such processing can rarely be done with one tool or framework — a number of tools are often involved, each having prowess in a particular field of the processing pipeline. In this session, we will see the latest endeavors of Apache Ignite to integrate with other big data platforms and provide its in-memory computing strengths for data processing pipelines. In particular we will have a closer look at how it can be integrated and used with Apache Kafka and/or Flume, and outline several use scenarios.

Event Streaming Architectures with Confluent and ScyllaDBScyllaDB

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!confluent

This document provides an overview and introduction to Apache Kafka and KSQL for building streaming data pipelines. It discusses how Kafka is an event streaming platform that can be used for messaging, streaming data, and stream processing. It then introduces KSQL, which is a streaming SQL engine for Apache Kafka that allows users to perform stream processing by writing SQL-like queries against Kafka topics. The document uses diagrams and examples to illustrate how to build a streaming data pipeline using Kafka Connect to ingest data, Kafka to store and transport streams, and KSQL to perform stream processing, enrichment, and analytics.

Kafka On YARN (KOYA): An Open Source Initiative to integrate Kafka & YARNDataWorks Summit

Kafka for data scientistsJenn Rawlins

Writing Blazing Fast, and Production-Ready Kafka Streams apps in less than 30...HostedbyConfluent

If you have already worked on various Kafka Streams applications before, then you have probably found yourself in the situation of rewriting the same piece of code again and again. Whether it's to manage processing failures or bad records, to use interactive queries, to organize your code, to deploy or to monitor your Kafka Streams app, build some in-house libraries to standardize common patterns across your projects seems to be unavoidable. And, if you're new to Kafka Streams you might be interested to know what are those patterns to use for your next streaming project. In this talk, I propose to introduce you to Azkarra, an open-source lightweight Java framework that was designed to provide most of that stuffs off-the-shelf by leveraging the best-of-breed ideas and proven practices from the Apache Kafka community.

Building streaming data applications using Kafka*[Connect + Core + Streams] b...Data Con LA

Abstract:- Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing. In this talk you will learn more about: A quick introduction to Kafka Core, Kafka Connect and Kafka Streams through code examples, key concepts and key features. A reference architecture for building such Kafka-based streaming data applications. A demo of an end-to-end Kafka-based streaming data application.

Connect, Test, Optimize: The Ultimate Kafka Connector Benchmarking ToolkitHostedbyConfluent

"Kafka Connect is quintessential for most production-grade Kafka data pipelines. As a core Kafka Connect team supporting more than 75 connectors at Confluent, we needed a platform that not only enables easy repeatability of tests across a breadth of connectors but also facilitates benchmarking and extensive analysis of diverse test runs. This session goes over the challenges in designing a framework to test connector performance. We will cover specific aspects of Connector testing, fine tuning parameters like connector configurations, clusters specifications for optimal performance. The framework integrates with various tools such as Jmeter, Hammerdb, ChaosMesh, Prometheus etc., scaled for Cloud-native environments. We will further go over how we optimized connector performance for maximum public record poll rates, cpu/memory usage, throughput, latencies for different data formats. Attendees learn in detail how real-world scenarios can be simulated in a test setup for a few connectors like S3 Sink Connector. We also go over some generic guidelines for performance testing of Kafka Connectors and practical considerations for implementing such test frameworks in general."

Jug - ecosystemFlorent Ramiere

This document provides an overview and agenda for a presentation on Confluent, streaming, and KSQL. The presentation includes: an introduction to Confluent and Apache Kafka; an explanation of why streaming platforms are useful; an overview of the Confluent Platform and its components; key concepts in streaming and Kafka; a demonstration of Kafka Streams, Kafka Connect, and KSQL; and resources for further information. The presentation aims to explain streaming concepts, demonstrate Confluent tools, and allow for a question and answer session.

Connecting Apache Kafka With Mule ESBJitendra Bafna

This document provides information on connecting Apache Kafka with Mule ESB. It discusses the key components of Apache Kafka including topics, producers, consumers, partitions, brokers and clusters. It also outlines some common use cases for Apache Kafka like messaging, website activity tracking, metrics, and log aggregation. The document then provides step-by-step instructions on setting up Zookeeper and Apache Kafka on Windows Server. It demonstrates how to create topics, install the Anypoint Kafka connector in Mule, and build Mule flows to integrate Apache Kafka as a producer and consumer. Code examples are provided for the Mule flows.

Building Streaming Data Applications Using Apache KafkaSlim Baltagi

Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing. In this talk you will learn more about: 1. A quick introduction to Kafka Core, Kafka Connect and Kafka Streams: What is and why? 2. Code and step-by-step instructions to build an end-to-end streaming data application using Apache Kafka

Chti jug - 2018-06-26Florent Ramiere

This document contains an agenda and overview of Confluent and streaming with Kafka. The agenda includes introductions to Confluent, streaming, KSQL, and a demo. Confluent is presented as the company founded by the creators of Apache Kafka to develop streaming platforms based on Kafka. Key concepts of streaming, the Confluent platform, and Kafka Streams, Kafka Connect, and KSQL are summarized. The document concludes with resources and time for questions.