Kafka Workshop

Oct 28, 20160 likes353 views

This document discusses Kafka and message buses. It describes synchronous vs asynchronous communication and how a message bus can be used for event aggregation, CQRS, microservices, and event sourcing. It then provides details on Apache Kafka, including its creation by LinkedIn, use by major companies, and basic workflow. Key components of the Kafka stack like Zookeeper, brokers, producers, consumers and topics are explained. The document ends with an example of launching a local Kafka cluster and producing and consuming messages.

Kafka & message bus
Robin GRAILLON & Alexandre ANDRÉ
12/10/2016

Synchronous
● Call
● Processing
○ Might be long
● Response

Asynchronous
● Call
● Response
● Post-Processing

Message Bus
● Events aggregation
● CQRS oriented
● µS oriented
● Event sourcing oriented
● Language agnostic
● Multiple implementations
○ RabbitMQ
○ Kafka
○ etc.

Apache Kafka
● LinkedIn creation
● Open sourced in 2011
● Filesystem oriented
● Wrote in Scala
● Highly scalable
● Used by big companies
○ LinkedIn
○ Netflix
○ Spotify
○ Meetic

LinkedIn statistics
● 800 billion m/day
○ 175 TB
● 13 million m/sec
○ 2.75 GB
● 1100 kafka instances
○ 60 clusters

Basic workflow
Producer 1
Producer 2
Producer 3
Producer X
Consumer 1
Consumer 2
Consumer 3
Consumer Y
Kafka Stack

Kafka stack
● Zookeeper : scalability, manager
● Kafka broker : kafka server instance
● Consumer : consumes events and do things
● Producer : produces events (like user as connected)
● topic : event name
● partition : way to split messages between brokers

Kafka partitions (1)
For one topic (let’s say “astronaut.connection”)

...but easily !
./console run
HELP
run:zk run zookeeper
kill:zk kill zookeeper
run:kk run kafka
kill:kk kill kafka
producer run producer on topic
consumer run consumer on topic
t:c create a topic
t:d delete a topic
t:l list topics
What do you want to do?

Step 1: launch Zookeeper!
What do you want to do? run:zk
Configuration file? [./zookeeper.properties]
Port? [2181]
./../bin/zookeeper-server-start.sh ./zookeeper.properties
What do you want to do?

Step 2: launch Kafka!
What do you want to do? run:kk

Step 3: list topics!
What do you want to do? t:l
Zookeeper host? [localhost:2181]
./../bin/kafka-topics.sh --list --zookeeper localhost:2181
What do you want to do?

Step 4: create a topic!
What do you want to do? t:c
Topic name? [test]
Zookeeper host? [localhost:2181]
Partitions? [1]
Replication factor? [1]
./../bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test
--partitions 1 --replication-factor 1
Created topic "test".
What do you want to do?

Step 5: list topics AGAIIIIN!
What do you want to do? t:l
Zookeeper host? [localhost:2181]
./../bin/kafka-topics.sh --list --zookeeper localhost:2181
test
What do you want to do?

Step 6: run a consumer!
New terminal !
$ ./console run
What do you want to do? consumer
Topic name? [test]
Zookeeper host? [localhost:2181]
From beginning? [1]
./../bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
--from-beginning

Step 7: generate data!
What do you want to do? g
Message? [I'm an astronaut!]
How many times? [1] 10000000
File? [data_10000000.txt]
10000000/10000000 [============================] 100%

Step 8: produce events!
Back to the other terminal
What do you want to do? p
Topic name? [test]
brokers? [localhost:9092]
data_10000000.txt
Dataset to use? [data_10000000.txt]
./../bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test <
./data/data_10000000.txt

GroupId: experiment 1
● 2 groupId
● 2 partitions
● 2 consumers

GroupId: experiment 2
● 1 groupId
● 2 consumer

Documentation/resources
● Kafka quick setup
https://kafka.apache.org/quickstart
● Kafka at LinkedIn
https://engineering.linkedin.com/kafka/running-kafka-scale
● Why zookeeper ?
https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka

The document outlines exercises for learning the ELK stack using Docker. It introduces Elasticsearch for data storage, Logstash for collecting and parsing logs, and Kibana for visualization. The exercises demonstrate setting up the environment, sending a test message to ELK, analyzing Shakespeare works, customizing the Kibana dashboard, using Grok filters to parse logs, and filtering Apache logs with Logstash.

Docker and FluentdSATOSHI TAGOMORI

This document discusses using Docker and Fluentd together for logging container architectures. It describes different patterns for aggregating logs from containers using Fluentd, including 1-level and 2-level aggregation. A new Docker logging driver called "fluentd" is announced that will directly send container logs to Fluentd. The document also provides information on the official Fluentd Docker image.

How to Shrink from 5 Tiers to 2 in a Multitier Microservices Architecturevsoshnikov

A typical microservices architecture has five tiers: You need a web proxy like NGINX to handle slow clients and to serve static files You need an application web server like Apache or Django to run your application logic You need a caching layer like Memcached or Redis to cache frequently accessed data You need a database proxy to handle sharding and replication, abstracting that stuff away from the application You need a database management system to store your data Five tiers – isn't it too many? Yeah, that's what we think too. So we've come up with an idea to shrink the whole bunch of tiers down to just a couple. The web proxy and database proxy are combined into one tier, now just a proxy. The application server, the cache, and the DBMS are packed into the other tier. Fewer tiers mean less problems and more bonuses like data consistency and better performance. In this session, I'll explain how to do this magic and how it works under the hood.

Fluentd v1 and RoadmapTreasure Data, Inc.

Fluentd v1 will focus on providing a stable and pluggable architecture for log forwarding and processing. Key features include a new configuration format that allows embedding Ruby code, improved filtering and labeling of events, and an error stream for handling errors. Other potential improvements include multi-processing support, zero downtime restarts, an actor framework, and JRuby/Windows compatibility. td-agent2 will upgrade the core libraries and use the new v1 configuration by default.

How bol.com makes sense of its logs, using the Elastic technology stack.Renzo Tomà

Bol.com uses the Elastic (ELK) stack to make sense of logs from over 1,600 servers and 500-600 million events per day. Key aspects of their system include: 1. Shipping JSON-formatted log events from sources like Apache, databases, and applications to Redis queues to allow multiple Logstash instances to process events in real-time without data loss. 2. Enriching log events with information like request IDs to correlate requests across services, and IP-to-role mappings to identify client roles. 3. Using Elasticsearch aggregations and transformations to generate a directed graph of service dependencies based on logs, to help understand their distributed architecture.

Writing Rust Command Line ApplicationsAll Things Open

Collect distributed application logging using fluentd (EFK stack)Marco Pas

This document discusses using Fluentd to collect distributed application logging in a containerized environment. It provides an overview of Fluentd, including its pluggable architecture and configuration. It then demonstrates capturing logging from Docker containers and HTTP services and storing the logs in MongoDB and Elasticsearch as part of the ELK stack. It shows filtering and parsing logs from a Spring Boot application. Finally, it discusses setting up Fluentd for high availability.

Kong in 1.x TerritoryThibault Charbonnier

Kong 1.0 reached maturity with the addition of L4 proxying, service mesh capabilities using mTLS, and blue/green upgrade paths. Kong 1.1 introduced declarative configuration using a YAML file for DB-less deployments and importing configurations into databases. It also added tags for filtering entities. Upcoming versions will focus on performance, gRPC support, and Kong Enterprise features while continuing to consolidate the platform.

Event machinealmeidaricardo

Event Machine is a library for Ruby that provides event-driven I/O. It uses the reactor pattern to achieve high scalability, performance, and stability. Event Machine powers the backends of many large sites by enabling asynchronous programming and handling many concurrent connections efficiently. It can be used with tools like Rails, Thin, Goliath, MongoDB, and RabbitMQ to build highly scalable web applications. Benchmarks show that using Event Machine can process many more requests per second than a traditional Rails app due to its asynchronous and non-blocking nature.

Cortex: Prometheus as a Service, One Year OnKausal

Presented by Tom Wilkie at PromCon 2017 With Speaker Notes: https://goo.gl/V9qGva At PromCon 2016, I presented "Project Frankenstein: A multitenant, horizontally scalable Prometheus as a service". It's now one year later, and lots has changed - not least the name! This talk will discuss what we've learnt running a Prometheus service for the past year, the architectural changes we made from the original design, and the improvements we've made to the Cortex user experience.

Scaling an ELK stack at bol.comRenzo Tomà

A presentation about the deployment of an ELK stack at bol.com At bol.com we use Elasticsearch, Logstash and Kibana in a logsearch system that allows our developers and operations people to easilly access and search thru logevents coming from all layers of its infrastructure. The presentations explains the initial design and its failures. It continues with explaining the latest design (mid 2014). Its improvements. And finally a set of tips are giving regarding Logstash and Elasticsearch scaling. These slides were first presented at the Elasticsearch NL meetup on September 22nd 2014 at the Utrecht bol.com HQ.

PharoDAYS 2015: Pharo Status - by Markus DenkerPharo

Pharo 4 will be released in Spring 2015 and contains many improvements and updates over previous versions. It has already seen over 1200 issues closed and is very stable. Small changes include improved refactorings, a smaller 6MB deployment image, and ifTrue: working on non-Booleans. Larger ongoing projects include first class variables, replacing the .changes system with Epicea, advanced reflection work, and VM improvements.

BSides London - Scapy WorkshopAdam Maxwell

This document provides an overview of a Scapy workshop that will cover topics including: - Installing and using Scapy to forge, decode, send, and receive packets - Creating basic packets like TCP, UDP, ICMP and modifying packet fields - Advanced packet crafting including HTTP requests and DNS queries - Using Scapy with Python for tasks like packet sniffing and parsing PCAP files - Replaying and filtering captured packets - The presenter's background and goals for the workshop

GeoDistributed datacenter: the DNS wayMoyd.co LTD

Alberto Zuin presents on moyd.co's geo-distributed infrastructure using OpenNebula. Moyd.co operates DNS servers across 4 datacenters in Europe and the US with anycast BGP routing. Each datacenter runs independent DNS and API servers with a shared MongoDB database. OpenNebula is used to federate the distributed infrastructure and configure DNS VMs through contextualization. Example use cases include distributing a news site, email servers, and log storage across multiple datacenters.

Ansible with AWSAllan Denot

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Puppet

8 Ways Network Engineers use Snabb (RIPE 77)Igalia

Fluentd v1 and future at techtalkN Masahiro

Fluentd v1 provides major improvements over v0.12 including nanosecond event time resolution, multi-core support, Windows support, and new plugin APIs. The new plugin APIs provide well-controlled lifecycles and integrate all output plugins. v1 also introduces a server engine based supervisor, dynamic buffering capabilities, and various plugin helpers. While maintaining compatibility with v0.12 plugins, v1 focuses on ease of use, stability, performance and flexibility.

Docker Workshop - Orchestrating Docker ContainersHugo Henley

This document outlines an agenda for a Docker workshop on orchestrating Docker containers. It discusses various tools in the container ecosystem including Rancher, Mesos, Marathon, service discovery with Consul and Registrator, monitoring with Prometheus and Grafana, and continuous integration/deployment with Jenkins. Mesos is presented as a way to share resources across frameworks like Marathon. Consul and Registrator enable service discovery so applications can be accessed regardless of where they run. Prometheus and Grafana provide monitoring. The workshop emphasizes combining these tools - Docker, Mesos, Marathon, Rancher, Consul, Registrator, Consul Template, cAdvisor, Prometheus, Grafana, and Jenkins - for a complete container management solution.

AGES Presentation on Web, Python, Django and GeoServerNg'eno Victor

Beyond Fault Tolerance with Actor ProgrammingFabio Tiriticco

Actor Programming is a software building approach that lets you can go beyond fault tolerance and achieve Resilience, which is the capacity of a system to self-heal and spring back into a fresh shape. First I'll introduce the difference between Reactive Programming and Reactive Systems, and then we'll go over a couple of implementation examples using Scala and Akka. The coupled GitHub repository with the code is here: https://github.com/ticofab/ActorDemo

Woo: Writing a fast web server @ ELS2015fukamachi

Eitaro Fukamachi presented on writing the fast web server Woo in Common Lisp. He discussed three tactics for achieving speed: using a multithreaded event-driven architecture instead of prefork, developing a fast HTTP parser instead of regular expressions, and using the libev event library. Benchmarks showed Woo was 1.6 times faster than Node.js. The goal was to create the fastest HTTP server in Common Lisp.

Алексей Петров "PHP at Scale: Knowing enough to be dangerous!"Fwdays

PHP at Scale: Knowing enough to be dangerous! by Oleksii Petrov discusses how to scale PHP applications. It covers strategies like caching, queueing, read/write splitting, and sharding. It also discusses using load balancers and choosing the right database. The key is to improve system metrics without dramatically changing the system. Scaling is predefined by your stack and architecture. Performance comes from optimizations everywhere, not just PHP. Being distributed is very challenging.

Ruby eventmachine pres at rubybdxMathieu Elie

Sphinx && Perl Houston Perl Mongers - May 8th, 2014Brett Estrade

The document discusses using Sphinx to index and search data. Sphinx allows indexing data from MySQL, PostgreSQL, and XMLPipe2 formats. It supports searching via its native protocol or MySQL protocol. The document provides examples of indexing over 2 million rows from a MariaDB database, filtering search results using hashes, transforming data to the XMLPipe2 format, and dynamically configuring indexes when data partitions change over time using the Sphinx::Config::Builder module. Real-world use cases discussed include providing a search interface for auction data and indexing multiple data partitions that can change.

Developing high-performance network servers in LispVladimir Sedach

Pharo Status Fosdem 2015Marcus Denker

Pharo 4 was released in Spring 2015, containing many improvements and updates since Pharo 3 in April 2014. These include improved refactorings, a dark theme, GT tools replacing old tools, first class variables, advanced reflection capabilities, Epicea replacing .changes, and a new GC called Spur. Future plans include a 64-bit COG VM, an optimizer called Sista, updated windowing with SDL2, a Block redo in Morphic, 3D with Woden, and a virtual GPU. Pharo remains very active with ongoing development and a welcoming community.

Writing Better Haskellnkpart

- The Kit dependency manager was initially written in one weekend with little Haskell knowledge and resulted in bad code with IO everywhere and little usage of standard libraries. - Over time, the code was improved by removing IO usage, learning to leverage standard libraries and Hackage packages, cleaning up imports, and specifying invariants through QuickCheck properties and tighter types. - While an improvement, the code was still not considered "good" and could be further refined by fully removing IO from the monad stack and defining more precise invariants.

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Netflix Keystone—Cloud scale event processing pipelineMonal Daxini

http://www.oreilly.com/pub/e/3764 Keystone processes over 700 billion events per day (1 peta byte) with at-least-once processing semantics in the cloud. Monal Daxini details how they used Kafka, Samza, Docker, and Linux at scale to implement a multi-tenant pipeline in AWS cloud within a year. He'll also share plans on offering a Stream Processing as a Service for all of Netflix use.

More Related Content

What's hot (20)

Event machinealmeidaricardo

Cortex: Prometheus as a Service, One Year OnKausal

Scaling an ELK stack at bol.comRenzo Tomà

PharoDAYS 2015: Pharo Status - by Markus DenkerPharo

BSides London - Scapy WorkshopAdam Maxwell

GeoDistributed datacenter: the DNS wayMoyd.co LTD

Ansible with AWSAllan Denot

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Puppet

8 Ways Network Engineers use Snabb (RIPE 77)Igalia

Fluentd v1 and future at techtalkN Masahiro

Docker Workshop - Orchestrating Docker ContainersHugo Henley

AGES Presentation on Web, Python, Django and GeoServerNg'eno Victor

Beyond Fault Tolerance with Actor ProgrammingFabio Tiriticco

Woo: Writing a fast web server @ ELS2015fukamachi

Алексей Петров "PHP at Scale: Knowing enough to be dangerous!"Fwdays

Ruby eventmachine pres at rubybdxMathieu Elie

Sphinx && Perl Houston Perl Mongers - May 8th, 2014Brett Estrade

Developing high-performance network servers in LispVladimir Sedach

Pharo Status Fosdem 2015Marcus Denker

Writing Better Haskellnkpart

Event machinealmeidaricardo

Cortex: Prometheus as a Service, One Year OnKausal

Scaling an ELK stack at bol.comRenzo Tomà

PharoDAYS 2015: Pharo Status - by Markus DenkerPharo

BSides London - Scapy WorkshopAdam Maxwell

GeoDistributed datacenter: the DNS wayMoyd.co LTD

Ansible with AWSAllan Denot

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014Puppet

8 Ways Network Engineers use Snabb (RIPE 77)Igalia

Fluentd v1 and future at techtalkN Masahiro

Docker Workshop - Orchestrating Docker ContainersHugo Henley

AGES Presentation on Web, Python, Django and GeoServerNg'eno Victor

Beyond Fault Tolerance with Actor ProgrammingFabio Tiriticco

Woo: Writing a fast web server @ ELS2015fukamachi

Алексей Петров "PHP at Scale: Knowing enough to be dangerous!"Fwdays

Ruby eventmachine pres at rubybdxMathieu Elie

Sphinx && Perl Houston Perl Mongers - May 8th, 2014Brett Estrade

Developing high-performance network servers in LispVladimir Sedach

Pharo Status Fosdem 2015Marcus Denker

Writing Better Haskellnkpart

Similar to Kafka Workshop (20)

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Netflix Keystone—Cloud scale event processing pipelineMonal Daxini

Final_Report_new (1)Adarsh Burma

This document outlines a log analysis project that streams logs from Kafka into Spark for analysis. It includes configurations for setting up a multi-broker Kafka cluster, integrating Kafka with the Spark application, and code snippets for streaming the logs. The Spark application then analyzes the logs and outputs statistics to an interactive web page, including top endpoints, frequent IP addresses, and response code counts. Screenshots show the web output and Spark UI during job execution.

Introduction to apache kafkaSamuel Kerrien

Apache KafkaJoe Stein

Apache Kafka is a distributed publish-subscribe messaging system that was originally created by LinkedIn and contributed to the Apache Software Foundation. It is written in Scala and provides a multi-language API to publish and consume streams of records. Kafka is useful for both log aggregation and real-time messaging due to its high performance, scalability, and ability to serve as both a distributed messaging system and log storage system with a single unified architecture. To use Kafka, one runs Zookeeper for coordination, Kafka brokers to form a cluster, and then publishes and consumes messages with a producer API and consumer API.

Introduction to Apache KafkaShiao-An Yuan

This document provides an introduction to Apache Kafka. It describes Kafka as a distributed messaging system with features like durability, scalability, publish-subscribe capabilities, and ordering. It discusses key Kafka concepts like producers, consumers, topics, partitions and brokers. It also summarizes use cases for Kafka and how to implement producers and consumers in code. Finally, it briefly outlines related tools like Kafka Connect and Kafka Streams that build upon the Kafka platform.

Building Stream Processing as a ServiceSteven Wu

This document discusses building stream processing as a service (SPaaS) using Apache Flink. It introduces Flink's stream processing capabilities and describes how to build a SPaaS offering with different levels of complexity and ease of use. It also covers the Keystone router for simple stream routing, building custom Flink jobs, and techniques for recovering from failures using backfill from Hive or rewinding the Flink job.

Fullstack workshopAssaf Gannon

Scio - Moving to Google Cloud, A Spotify StoryNeville Li

Talk at Philly ETE Apr 28 2017 We will talk about Spotify’s story of migrating our big data infrastructure to Google Cloud. Over the past year or so we moved away from maintaining our own 2500+ node Hadoop cluster to managed services in the cloud. We replaced two key components in our data processing stack, Hive and Scalding, with BigQuery and Scio and are able to iterate at a much faster speed. We will focus the technical aspect of Scio, a Scala API for Apache Beam and Google Cloud Dataflow and how it changed the way we process data.

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015Monal Daxini

Keystone - Processing over Half a Trillion events per day with 8 million events & 17 GB per second peaks, and at-least once processing semantics. We will explore in detail how we employ Kafka, Samza, and Docker at scale to implement a multi-tenant pipeline. We will also look at the evolution to its current state and where the pipeline is headed next in offering a self-service stream processing infrastructure atop the Kafka based pipeline and support Spark Streaming.

[KubeCon EU 2020] containerd Deep DiveAkihiro Suda

Ansiblefest 2018 Network automation journey at robloxDamien Garros

In December 2017, Roblox’s network was managed in a traditional way without automation. To sustained its growth, the team had to deploy 2 datacenters, a global network and multiple point of presence around the world in few months, the only solution to be able to achieve that was to automate everything. 6 months later, the team has made tremendous progress and many aspects of the network lifecycle has been automated from the routers, switches to the load balancers. Synopsis This talk is a retrospective of Roblox’s journey into Network automation: How we got started and how we automated an existing network. How we organized the project around Github and an DCIM/IPAM solution (netbox), How Docker helped us to package Ansible and create a consistent environment. How we managed many roles and variations of our design in single project How we have automated the provisioning of our F5 Load Balancers. For each point, we’ll cover what was successful, what was more challenging and what limitations we had to deal with.

04-Kafka.pptxMannMehta13

Kafka is a fast, scalable, and durable distributed commit log service that can handle millions of messages per second. It is used by many large companies as a messaging backbone and for real-time data processing. Kafka topics are partitioned and replicated for high throughput and fault tolerance. Producers write messages to topics that are consumed by consumer groups who track their progress using offsets.

04-Kafka.pptxAdityaGanguly12

10 Lessons Learned from using Kafka with 1000 microservices - java global summitNatan Silnitsky

Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices. We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place. In these slides you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include: * How to increase dev velocity of event driven style code. * How to optimize working with Kafka in polyglot setting * How to support growing amount of traffic and developers. * How to tackle multiple DCs environment.

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky

At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker. This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters. In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner. You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant. You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC. When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.

Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent

"As Apache Kafka gains widespread adoption, an increasing number of people face its pitfalls. Despite completing courses and reading documentation, many encounter hurdles navigating Kafka's subtle complexities. Join us for an enlightening session led by the customer support team of Conduktor, where we engage daily with users grappling with Kafka's subtleties. We've observed recurring themes in user queries: What happens when a consumer group rebalances? What is an advertised listener? Why aren't my records displayed in chronological order when I consume them? How does retention work? For all these questions, the answer is ""It depends"". In this talk, we aim to demystify these uncertainties by presenting nuanced scenarios for each query. That way you will be more confident on how your Kafka infrastructure works behind the scenes, and you'll be equipped to share this knowledge with your colleagues. By being aware of the most common misconceptions, you should be able to both speed up your own learning curve and also help others more effectively."

DockerCon 2022 - From legacy to Kubernetes, securely & quicklyEric Smalling

You’ve been developing software for years and now your team is ready to take the plunge into orchestrated containers and Kubernetes. You’ve learned about containers, images, and Dockerfiles, but standing up a Kubernetes cluster and actually running your app in it seems like a daunting task. In this session, we’ll go over the basics to get your app up and running in Kubernetes right on your own workstation using Docker Desktop. On the way, we’ll cover some of the security aspects you need to keep in mind and show you how to implement them in your Kubernetes manifests. We’ll go over: 1.) Kubernetes basics, including pods, deployments, and services 2.) Moving a legacy app into a container and running it in Kubernetes 3.) Some security best practices to watch out for — and what can happen if you don’t 4.) Implementing those best practices to defend against and limit the blast radius of an attack

Enabling Data Scientists to easily create and own Kafka ConsumersStefan Krawczyk

At Stitch Fix, we hire Full Stack Data Scientists (145+) and expect them to perform diverse functions: from conception to modeling to implementation to measurement. Since Kafka is the way we get event data, this inevitably means that a Data Scientist will need to write a Kafka consumer if they’re going to complete their implementation work. E.g. to transform some client data into features, or perform a model prediction, or allocate someone to an A/B test, etc. In this talk I’ll go over how we built an opinionated Kafka client to easily enable Data Scientists to deploy and own production Kafka consumers, by focusing on writing python functions rather than fighting pitfalls with Kafka.

Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...HostedbyConfluent

The document summarizes an expert talk given by Stefan Krawczyk on enabling data scientists at Stitch Fix to easily create and own Kafka consumers. It discusses Stitch Fix as a company, data science at Stitch Fix, their opinionated Kafka consumer code, and learnings. It describes how data scientists are responsible for full stack data science, from idea to production. It provides an example of a simple "hello world" Kafka consumer in Python and explains the code. It outlines the differences between platform concerns owned by engineers and data science concerns.

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Netflix Keystone—Cloud scale event processing pipelineMonal Daxini

Final_Report_new (1)Adarsh Burma

Introduction to apache kafkaSamuel Kerrien

Apache KafkaJoe Stein

Introduction to Apache KafkaShiao-An Yuan

Building Stream Processing as a ServiceSteven Wu

Fullstack workshopAssaf Gannon

Scio - Moving to Google Cloud, A Spotify StoryNeville Li

Netflix Keystone Pipeline at Big Data Bootcamp, Santa Clara, Nov 2015Monal Daxini

[KubeCon EU 2020] containerd Deep DiveAkihiro Suda

Ansiblefest 2018 Network automation journey at robloxDamien Garros

04-Kafka.pptxMannMehta13

04-Kafka.pptxAdityaGanguly12

10 Lessons Learned from using Kafka with 1000 microservices - java global summitNatan Silnitsky

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky

Apache Kafka's Common Pitfalls & Intricacies: A Customer Support PerspectiveHostedbyConfluent

DockerCon 2022 - From legacy to Kubernetes, securely & quicklyEric Smalling

Enabling Data Scientists to easily create and own Kafka ConsumersStefan Krawczyk

Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...HostedbyConfluent

Recently uploaded (20)

ADVXAI IN MALWARE ANALYSIS FRAMEWORK: BALANCING EXPLAINABILITY WITH SECURITYijscai

With the increased use of Artificial Intelligence (AI) in malware analysis there is also an increased need to understand the decisions models make when identifying malicious artifacts. Explainable AI (XAI) becomes the answer to interpreting the decision-making process that AI malware analysis models use to determine malicious benign samples to gain trust that in a production environment, the system is able to catch malware. With any cyber innovation brings a new set of challenges and literature soon came out about XAI as a new attack vector. Adversarial XAI (AdvXAI) is a relatively new concept but with AI applications in many sectors, it is crucial to quickly respond to the attack surface that it creates. This paper seeks to conceptualize a theoretical framework focused on addressing AdvXAI in malware analysis in an effort to balance explainability with security. Following this framework, designing a machine with an AI malware detection and analysis model will ensure that it can effectively analyze malware, explain how it came to its decision, and be built securely to avoid adversarial attacks and manipulations. The framework focuses on choosing malware datasets to train the model, choosing the AI model, choosing an XAI technique, implementing AdvXAI defensive measures, and continually evaluating the model. This framework will significantly contribute to automated malware detection and XAI efforts allowing for secure systems that are resilient to adversarial attacks.

some basics electrical and electronics knowledgenguyentrungdo88

Compiler Design Unit1 PPT Phases of Compiler.pptxRushaliDeshmukh2

Lidar for Autonomous Driving, LiDAR Mapping for Driverless Cars.pptxRishavKumar530754

Smart_Storage_Systems_Production_Engineering.pptxrushikeshnavghare94

211421893-M-Tech-CIVIL-Structural-Engineering-pdf.pdfinmishra17121973

Data Structures_Introduction to algorithms.pptxRushaliDeshmukh2

Concept of Problem Solving, Introduction to Algorithms, Characteristics of Algorithms, Introduction to Data Structure, Data Structure Classification (Linear and Non-linear, Static and Dynamic, Persistent and Ephemeral data structures), Time complexity and Space complexity, Asymptotic Notation - The Big-O, Omega and Theta notation, Algorithmic upper bounds, lower bounds, Best, Worst and Average case analysis of an Algorithm, Abstract Data Types (ADT)

Smart Storage Solutions.pptx for production engineeringrushikeshnavghare94

Raish Khanji GTU 8th sem Internship Report.pdfRaishKhanji

This report details the practical experiences gained during an internship at Indo German Tool Room, Ahmedabad. The internship provided hands-on training in various manufacturing technologies, encompassing both conventional and advanced techniques. Significant emphasis was placed on machining processes, including operation and fundamental understanding of lathe and milling machines. Furthermore, the internship incorporated modern welding technology, notably through the application of an Augmented Reality (AR) simulator, offering a safe and effective environment for skill development. Exposure to industrial automation was achieved through practical exercises in Programmable Logic Controllers (PLCs) using Siemens TIA software and direct operation of industrial robots utilizing teach pendants. The principles and practical aspects of Computer Numerical Control (CNC) technology were also explored. Complementing these manufacturing processes, the internship included extensive application of SolidWorks software for design and modeling tasks. This comprehensive practical training has provided a foundational understanding of key aspects of modern manufacturing and design, enhancing the technical proficiency and readiness for future engineering endeavors.

Artificial Intelligence (AI) basics.pptxaditichinar

Introduction to FLUID MECHANICS & KINEMATICSnarayanaswamygdas

Fluid mechanics is the branch of physics concerned with the mechanics of fluids (liquids, gases, and plasmas) and the forces on them. Originally applied to water (hydromechanics), it found applications in a wide range of disciplines, including mechanical, aerospace, civil, chemical, and biomedical engineering, as well as geophysics, oceanography, meteorology, astrophysics, and biology. It can be divided into fluid statics, the study of various fluids at rest, and fluid dynamics. Fluid statics, also known as hydrostatics, is the study of fluids at rest, specifically when there's no relative motion between fluid particles. It focuses on the conditions under which fluids are in stable equilibrium and doesn't involve fluid motion. Fluid kinematics is the branch of fluid mechanics that focuses on describing and analyzing the motion of fluids, such as liquids and gases, without considering the forces that cause the motion. It deals with the geometrical and temporal aspects of fluid flow, including velocity and acceleration. Fluid dynamics, on the other hand, considers the forces acting on the fluid. Fluid dynamics is the study of the effect of forces on fluid motion. It is a branch of continuum mechanics, a subject which models matter without using the information that it is made out of atoms; that is, it models matter from a macroscopic viewpoint rather than from microscopic. Fluid mechanics, especially fluid dynamics, is an active field of research, typically mathematically complex. Many problems are partly or wholly unsolved and are best addressed by numerical methods, typically using computers. A modern discipline, called computational fluid dynamics (CFD), is devoted to this approach. Particle image velocimetry, an experimental method for visualizing and analyzing fluid flow, also takes advantage of the highly visual nature of fluid flow. Fundamentally, every fluid mechanical system is assumed to obey the basic laws : Conservation of mass Conservation of energy Conservation of momentum The continuum assumption For example, the assumption that mass is conserved means that for any fixed control volume (for example, a spherical volume)—enclosed by a control surface—the rate of change of the mass contained in that volume is equal to the rate at which mass is passing through the surface from outside to inside, minus the rate at which mass is passing from inside to outside. This can be expressed as an equation in integral form over the control volume. The continuum assumption is an idealization of continuum mechanics under which fluids can be treated as continuous, even though, on a microscopic scale, they are composed of molecules. Under the continuum assumption, macroscopic (observed/measurable) properties such as density, pressure, temperature, and bulk velocity are taken to be well-defined at "infinitesimal" volume elements—small in comparison to the characteristic length scale of the system, but large in comparison to molecular length scale

Degree_of_Automation.pdf for Instrumentation and industrial specialistshreyabhosale19

Level 1-Safety.pptx Presentation of Electrical SafetyJoseAlbertoCariasDel

Oil-gas_Unconventional oil and gass_reseviours.pdfM7md3li2

fluke dealers in bangalore..............Haresh Vaswani

The Fluke 925 is a vane anemometer, a handheld device designed to measure wind speed, air flow (volume), and temperature. It features a separate sensor and display unit, allowing greater flexibility and ease of use in tight or hard-to-reach spaces. The Fluke 925 is particularly suitable for HVAC (heating, ventilation, and air conditioning) maintenance in both residential and commercial buildings, offering a durable and cost-effective solution for routine airflow diagnostics.

QA/QC Manager (Quality management Expert)rccbatchplant

railway wheels, descaling after reheating and before forgingJavad Kadkhodapour

ELectronics Boards & Product Testing_Shiju.pdfShiju Jacob

introduction to machine learining for beginersJoydebSheet

Structural Response of Reinforced Self-Compacting Concrete Deep Beam Using Fi...Journal of Soft Computing in Civil Engineering

Analysis of reinforced concrete deep beam is based on simplified approximate method due to the complexity of the exact analysis. The complexity is due to a number of parameters affecting its response. To evaluate some of this parameters, finite element study of the structural behavior of the reinforced self-compacting concrete deep beam was carried out using Abaqus finite element modeling tool. The model was validated against experimental data from the literature. The parametric effects of varied concrete compressive strength, vertical web reinforcement ratio and horizontal web reinforcement ratio on the beam were tested on eight (8) different specimens under four points loads. The results of the validation work showed good agreement with the experimental studies. The parametric study revealed that the concrete compressive strength most significantly influenced the specimens’ response with the average of 41.1% and 49 % increment in the diagonal cracking and ultimate load respectively due to doubling of concrete compressive strength. Although the increase in horizontal web reinforcement ratio from 0.31 % to 0.63 % lead to average of 6.24 % increment on the diagonal cracking load, it does not influence the ultimate strength and the load-deflection response of the beams. Similar variation in vertical web reinforcement ratio leads to an average of 2.4 % and 15 % increment in cracking and ultimate load respectively with no appreciable effect on the load-deflection response.