Connecting Kafka Across Multiple AWS VPCs

Oct 23, 20185 likes4,730 views

(Benoit Carrière, Expedia) Kafka Summit SF 2018 As Expedia, the world’s largest online travel agency, moved to a multi Virtual Private Cloud (VPC) strategy in AWS, we faced the challenge of making our systems accessible, or using other systems, across many VPCs. In most cases, a secure internet-facing endpoint or doing VPC Peering should do the work, right? But what if the system isn’t a typical HTTP-based microservice? What if it’s a distributed, partitioned and binary protocol-based system, where anyone talks to everyone all the time? That’s exactly what we encountered when we tried to make our Kafka accessible to our clients. We solved this problem by leveraging Apache Kafka®‘s distributive nature, using AWS’ new VPC Endpoint technology and their recent Network Load Balancer, some Route53 records and a bit of creativity! In this session, I’ll dive into: -Our use case: Kafka accessible to other VPCs -Why we didn’t go with internet-facing endpoint or use VPC Peering -A brief description on how VPC endpoints work -Our solution to the problem: That’s where the fun starts.

Connecting Kafka Across Multiple
AWS VPCs
Benoit Carrière, Expedia Group

• Our use case

• Alternatives

• Our solution
Topics

Use Case
• Expedia site

• Hotel pages and +

• Diﬀerent AWS accounts/VPCs

• IP spaces overlapping issue

• Exposing more than what is
needed

• Too many peering to manage 
(one per client VPCs)

• Strongly discouraged by our
architects
VPC Peering ?
Can
I haz topics,
plz?
Can I haz
topics, plz?
Can I haz
topics, plz?
Can I haz
topics, plz?
Help
:(

Internet-Facing ?
• Judged to be too risky

• Had to manage whitelist of private IPs

• Firewall limitation (4 Gbps cap)

• Traﬃc going out of AWS private network

• Data transfer cost incurred

• Require redesign

• Payload size limit

• Retention limit

• Not a drop-in replacement for us
Kinesis ?
Robert Couse-Baker (CC-BY-2.0)

AWS VPC Endpoint
• Released November 2017

• PrivateLink service in a VPC

• Connect through an interface VPC endpoint

• Use NLB to expose service

• Bootstrap servers 
 
SSL://kafka.usw2.exp.com:6000
• Security settings 
 
security.protocol=SSL 
ssl.enabled.protocols=TLSv1.2 
ssl.protocol=TLSv1.2 
ssl.truststore.location=truststore.jks 
ssl.truststore.password=... 
ssl.keystore.location=keystore.jks 
ssl.keystore.password=... 
ssl.key.password=...
Client Side

• kafka.usw2.exp.com

• Route 53

• CNAME

• Hosted Zone
Client Side

• SSL://kafka.usw2.exp.com:6000

• Only one server?

• All brokers behind Load Balancer

• Security

• SSL/TLS certiﬁcates

• Two-way authentication
Client Side

Client Side
Finding Nemo - Pixar Animation Studios via Know Your Meme ®
kafka.usw2.exp.com
???

• Powered by PrivateLink

• Essentially exposes a NLB to another VPC

• Access managed using IAM principals

• DNS hostname 
 
vpce-afghnkxcptx0oefx0-98a1tpfp.vpce-
svc-jw9zwsubz7cbnxkg.us-
west-2.vpce.amazonaws.com

• https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html
AWS VPC Endpoint

kafka.usw2.exp.com
vpce-
afghnkxcptx0oefx0-98a1tpfp.v
pce-svc-jw9zwsubz7cbnxkg.us-
west-2.vpce.amazonaws.com
Client Side

• advertised.listeners
• Listeners for clients to use

• Diﬀerent from the interface to which the broker
binds (0.0.0.0)

• CNAME : kafka.usw2.exp.com

• Unique network port per broker

• 6001, 6002, etc.
Server Side

• Target Groups

• One per broker

• One partition leader at any time

• Group 6000 for bootstrapping
Server Side

• Network Load Balancer

• Act as a reverse proxy

• Can’t balance request

• One leader per partition

• Cross-Zone load balancing

• One-to-one mapping

• Listener ➞ Group ➞ Instance
Server Side

• Example

Listener 6001

Target Group 6001

EC2 Instance for broker 6001

• Keep things aligned!
Server Side

1. VPC Endpoint Service - Provider 
 
2. VPC Endpoint - Consumer 
 
3. Consumer requests connection 
 
4. Provider accepts connection
VPC Endpoint

SSL://...:6000SSL://...:6002
Broker
6002 is leader
on partition
topic-0.
Roger
that.

And it works !
Simple !
Activity on the VPC Endpoint NLB
Topics size progression during data load

• Kafka over VPC Endpoint

• No VPC Peering, no Internet-facing

• Secure, resilient, eﬃcient

• Takeaways

• VPC Endpoint can work for distributed systems

• NLB as a reverse proxy

• Kafka + NLB = unique broker endpoint
Conclusion

Connecting Kafka Across Multiple AWS VPCs

Containers are everywhere. But what exactly is a container? What are they made from? What's the difference between LXC, butts-nspawn, Docker, and the other container systems out there? And why should we bother about specific filesystems? In this talk, Jérôme will show the individual roles and behaviors of the components making up a container: namespaces, control groups, and copy-on-write systems. Then, he will use them to assemble a container from scratch, and highlight the differences (and likelinesses) with existing container systems.

[KubeCon EU 2022] Running containerd and k3s on macOSAkihiro Suda

https://sched.co/ytpi It has been very hard to use Mac for developing containerized apps. A typical way is to use Docker for Mac, but it is not FLOSS. Another option is to install Docker and/or Kubernetes into VirtualBox, often via minikube, but it doesn't propagate localhost ports, and VirtualBox also doesn't support the ARM architecture. This session will show how to run containerd and k3s on macOS, using Lima and Rancher Desktop. Lima wraps QEMU in a simple CLI, with neat features for container users, such as filesystem sharing and automatic localhost port forwarding, as well as DNS and proxy propagation for enterprise networks. Rancher Desktop wraps Lima with k3s integration and GUI.

Docker Networking Deep DiveDocker, Inc.

This document provides an overview and agenda for a Docker networking deep dive presentation. The presentation covers key concepts in Docker networking including libnetwork, the Container Networking Model (CNM), multi-host networking capabilities, service discovery, load balancing, and new features in Docker 1.12 like routing mesh and secured control/data planes. The agenda demonstrates Docker networking use cases like default bridge networks, user-defined bridge networks, and overlay networks. It also covers networking drivers, Docker 1.12 swarm mode networking functionality, and how concepts like routing mesh and load balancing work.

Tuning Apache Kafka Connectors for Flink.pptxFlink Forward

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Designing High Availability for HashiCorp Vault in AWS☁ Bryan Krausen

Red team upgrades using sccm for malware deploymentenigma0x3

The document discusses using System Center Configuration Manager (SCCM) to deploy malware laterally across an enterprise network. Specifically, it describes how an attacker with access to an SCCM server could (1) create packages containing PowerShell payloads, (2) deploy those packages to targeted groups of machines, and (3) rely on SCCM agents to fetch and execute the payloads without generating additional network traffic. The document argues that abusing SCCM in this way allows for strategic, targeted malware deployment while blending in with normal enterprise traffic.

Terraform Introductionsoniasnowfrog

This document discusses the infrastructure provisioning tool Terraform. It can be used to provision resources like EC2 instances, storage, and DNS entries across multiple cloud providers. Terraform uses configuration files to define what infrastructure should be created and maintains state files to track changes. It generates execution plans to determine what changes need to be made and allows applying those changes to create, update or destroy infrastructure.

Kvm and libvirtplarsen67

LibVirt and KVM provide virtualization capabilities on Linux systems. LibVirt uses a standardized API to manage different hypervisors like KVM. KVM allows running virtual machines at native speeds by using hardware virtualization extensions. It provides high density and portability compared to running systems directly on hardware. LibVirt handles tasks like networking, storage, and interfaces through tools like Virsh and Virt-Manager, and advanced topics include security with SELinux, bridged networking, and remote access.

SouJava May 2020: Apache Camel 3 - the next generation of enterprise integrationClaus Ibsen

In this session, we'll discuss: - What’s Apache Camel: An overview of Camel and what you use it for and why you should care. - Camel 3: Demos of how Camel 3, Camel K and Camel Quarkus all work together, and will provide insights into Camel’s role in the next major release of Red Hat Integration products. - Camel K: This serverless integration platform provides low-code/no-code capabilities, where integrations can be snapped together quickly using the powers from integration patterns and Camel’s extensive set of connectors. - Camel Quarkus: Using Knative (the fast runtime of Quarkus) and Camel K brings awesome serverless features, such as auto-scaling, scaling to zero, and event-based communication, with great integration capabilities from Apache Camel. You will also hear about the latest Camel sub-project Camel Kafka Connectors which makes it possible to use all the Camel components as Kafka Connect connectors. Finally we bring details of the roadmap for what is coming up in the Camel projects. And after the presentation we have about 30 minutes of QA answering all the questions from the audience.

Introduction to Apache ZooKeeperSaurav Haloi

Fuzzing malware for fun & profit. Applying Coverage-Guided Fuzzing to Find Bu...Maksim Shudrak

Docker introductiondotCloud

Docker is a system for running applications in isolated containers. It addresses issues with traditional virtual machines by providing lightweight containers that share resources and allow applications to run consistently across different environments. Docker eliminates inconsistencies in development, testing and production environments. It allows applications and their dependencies to be packaged into a standardized unit called a container that can run on any Linux server. This makes applications highly portable and improves efficiency across the entire development lifecycle.

MAAS High Availability OverviewChristian "kiko" Reis

OpenStack networking juno l3 h-a, dvrSim Janghoon

Namespaces and cgroups - the basis of Linux containersKernel TLV

Agenda: * Background: namespaces/cgroups, the basis for container virtualization * Namespace implementation * The 6 kernel namespaces - some implementation details. * System calls for namespaces * Usage examples * cgroups kernel implementation * cgroup VFS * cgroup filesystem operations for handling cgroups examples * The cgroup implementation + some userspace examples * Checkpoint/Restore in brief

High Availability Content Caching with NGINXNGINX, Inc.

On-Demand Recording: https://www.nginx.com/resources/webinars/high-availability-content-caching-nginx/ You trust NGINX to be your web server, but did you know it’s also a high-performance content cache? In fact, the world’s most popular CDNs – CloudFlare, MaxCDN, and Level 3 among them – are built on top of the open source NGINX software. NGINX content caching can drastically improve the performance of your applications. We’ll start with basic configuration, then move on to advanced concepts and best practices for architecting high availability and capacity in your application infrastructure. Join this webinar to: * Enable content caching with the key configuration directives * Use micro caching with NGINX Plus to cache dynamic content while maintaining low CPU utilization * Partition your cache across multiple servers for high availability and increased capacity * Log transactions and troubleshoot your NGINX content cache

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Henning Jacobs

Talk given at JAX DevOps London on 2019-05-15. Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 90+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are open source and can be applied to most Kubernetes deployments. Topics covered in the talk include: understanding resource requests and limits, cgroups and CFS quota behavior, contributing factors to cluster costs (in public clouds), and best practices for managing Kubernetes resources.

Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...Jérôme Petazzoni

Terraform modules and best-practices - September 2018Anton Babenko

Complete-NGINX-Cookbook-2019.pdfTomaszWojciechowski22

TerraformDiego Pacheco

This document discusses infrastructure as code and the HashiCorp ecosystem. Infrastructure as code allows users to define and provision infrastructure through code rather than manual configuration. It can be used to launch, create, change, and downscale infrastructure based on configuration files. Tools like Terraform allow showing what changes will occur before applying them through files like main.tf and variables.tf. Terraform is part of the broader HashiCorp ecosystem of tools.

Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17Gwen (Chen) Shapira

This document discusses disaster recovery strategies for Apache Kafka clusters running across multiple data centers. It outlines several failure scenarios like an entire data center being demolished and recommends solutions like running a single Kafka cluster across multiple near-by data centers. It then describes a "stretch cluster" approach using 3 data centers with replication between them to provide high availability. The document also discusses active-active replication between two data center clusters and challenges around consumer offsets not being identical across data centers during a failover. It recommends approaches like tracking timestamps and failing over consumers based on time.

TerraformPathum Fernando ☁

This document provides an overview and introduction to Terraform, including: - Terraform is an open-source tool for building, changing, and versioning infrastructure safely and efficiently across multiple cloud providers and custom solutions. - It discusses how Terraform compares to other tools like CloudFormation, Puppet, Chef, etc. and highlights some key Terraform facts like its versioning, community, and issue tracking on GitHub. - The document provides instructions on getting started with Terraform by installing it and describes some common Terraform commands like apply, plan, and refresh. - Finally, it briefly outlines some key Terraform features and example use cases like cloud app setup, multi

HDFS on Kubernetes—Lessons Learned with Kimoon KimDatabricks

There is growing interest in running Apache Spark natively on Kubernetes (see https://github.com/apache-spark-on-k8s/spark). Spark applications often access data in HDFS, and Spark supports HDFS locality by scheduling tasks on nodes that have the task input data on their local disks. When running Spark on Kubernetes, if the HDFS daemons run outside Kubernetes, applications will slow down while accessing the data remotely. This session will demonstrate how to run HDFS inside Kubernetes to speed up Spark. In particular, it will show how Spark scheduler can still provide HDFS data locality on Kubernetes by discovering the mapping of Kubernetes containers to physical nodes to HDFS datanode daemons. You’ll also learn how you can provide Spark with the high availability of the critical HDFS namenode service when running HDFS in Kubernetes.

Linux kernel tracingViller Hsiao

This document discusses tracing in the Linux kernel. It describes various tracing mechanisms like ftrace, tracepoints, kprobes, perf, and eBPF. Ftrace allows tracing functions via compiler instrumentation or dynamically. Tracepoints define custom trace events that can be inserted at specific points. Kprobes and related probes like jprobes allow tracing kernel functions. Perf provides performance monitoring capabilities. eBPF enables custom tracing programs to be run efficiently in the kernel via just-in-time compilation. Tracing tools like perf, systemtap, and LTTng provide user interfaces.

Docker, LinuX ContainerAraf Karsh Hamid

Advanced TerraformSamsung Electronics

Kernel Recipes 2019 - Faster IO through io_uringAnne Nicolas

io_uring provides a new asynchronous I/O interface in Linux that aims to address limitations with existing interfaces like aio and libaio. It uses a ring-based model for submission and completion queues to efficiently support asynchronous I/O operations with low latency and high throughput. Though initially skeptical, Linus Torvalds ultimately merged io_uring into the Linux kernel due to improvements in missing features, ease of use, and efficiency over alternatives.

OpenStack and WindowsAlessandro Pilotti

OpenStack is an open source cloud computing platform that can manage large networks of virtual machines and physical servers. It uses a distributed architecture with components like Nova (compute), Swift (object storage), Cinder (block storage), and Quantum (networking). OpenStack has been successful due to its scalability, support for multiple hypervisors including Hyper-V, and compatibility with popular programming languages like Python. While OpenStack is best suited for large public and private clouds, its complex installation and lack of unified deployment tools can present challenges, especially for small to mid-sized clouds.

"Turning Kubernetes into a full-fledged private cloud", Volodymyr TsapFwdays

At the conference, I will present an approach to extending the capabilities of Kubernetes, transforming it into a fully-fledged private Cloud with support for virtualization, isolated networks, and multi-organization authentication. The goal of the presentation is to demonstrate that Kubernetes is not just a platform for containers, but a fully-featured private cloud that can be easily adapted to business needs.

More Related Content

What's hot (20)

SouJava May 2020: Apache Camel 3 - the next generation of enterprise integrationClaus Ibsen

Introduction to Apache ZooKeeperSaurav Haloi

Fuzzing malware for fun & profit. Applying Coverage-Guided Fuzzing to Find Bu...Maksim Shudrak

Docker introductiondotCloud

MAAS High Availability OverviewChristian "kiko" Reis

OpenStack networking juno l3 h-a, dvrSim Janghoon

Namespaces and cgroups - the basis of Linux containersKernel TLV

High Availability Content Caching with NGINXNGINX, Inc.

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Henning Jacobs

Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...Jérôme Petazzoni

Terraform modules and best-practices - September 2018Anton Babenko

Complete-NGINX-Cookbook-2019.pdfTomaszWojciechowski22

TerraformDiego Pacheco

Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17Gwen (Chen) Shapira

TerraformPathum Fernando ☁

HDFS on Kubernetes—Lessons Learned with Kimoon KimDatabricks

Linux kernel tracingViller Hsiao

Docker, LinuX ContainerAraf Karsh Hamid

Advanced TerraformSamsung Electronics

Kernel Recipes 2019 - Faster IO through io_uringAnne Nicolas

SouJava May 2020: Apache Camel 3 - the next generation of enterprise integrationClaus Ibsen

Introduction to Apache ZooKeeperSaurav Haloi

Fuzzing malware for fun & profit. Applying Coverage-Guided Fuzzing to Find Bu...Maksim Shudrak

Docker introductiondotCloud

MAAS High Availability OverviewChristian "kiko" Reis

OpenStack networking juno l3 h-a, dvrSim Janghoon

Namespaces and cgroups - the basis of Linux containersKernel TLV

High Availability Content Caching with NGINXNGINX, Inc.

Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Henning Jacobs

Cgroups, namespaces, and beyond: what are containers made from? (DockerCon Eu...Jérôme Petazzoni

Terraform modules and best-practices - September 2018Anton Babenko

Complete-NGINX-Cookbook-2019.pdfTomaszWojciechowski22

TerraformDiego Pacheco

Multi-Cluster and Failover for Apache Kafka - Kafka Summit SF 17Gwen (Chen) Shapira

TerraformPathum Fernando ☁

HDFS on Kubernetes—Lessons Learned with Kimoon KimDatabricks

Linux kernel tracingViller Hsiao

Docker, LinuX ContainerAraf Karsh Hamid

Advanced TerraformSamsung Electronics

Kernel Recipes 2019 - Faster IO through io_uringAnne Nicolas

Similar to Connecting Kafka Across Multiple AWS VPCs (20)

OpenStack and WindowsAlessandro Pilotti

"Turning Kubernetes into a full-fledged private cloud", Volodymyr TsapFwdays

Directions for CloudStack NetworkingChiradeep Vittal

The Future of SDN in CloudStack by Chiradeep Vittalbuildacloud

The core of CloudStack networking has always been software-defined. As the networking industry evolves to a software-defined future, CloudStack will have to evolve with it. The presentation will examine the present state of SDN in CloudStack, look at some industry directions and attempt to predict the evolution of CloudStack with those trends. Bio Chiradeep Vittal is a Distinguished Engineer in the Converged Infrastructure Group at Citrix where he has technology leadership responsibilities around Citrix Cloud Platform, Citrix Lifecycle Manager and Citrix Workspace Pod. He is also a Project Management Committee member of the Apache CloudStack Project. At cloud.com (acquired by Citrix), he was a founding engineer, often tasked with the thorny details of virtualized networking and storage. Prior to cloud.com, he worked at several Silicon Valley startups in various architectural roles. Chiradeep has a B.Tech in Computer Science from IIT, Bombay and a M.Sc from the University of Alberta. He has spoken / presented at several conferences, including CloudStack Collab, LISA, OSCON, ONS, SDN Summit and LinuxCon. His twitter handle is @chiradeep and occasionally blogs at http://cloudierthanthou.wordpress.com

Hacking apache cloud stackNitin Mehta

Service Discovery: From Classic to VPCMark Corwin

As SeatGeek's traffic continues to grow, so too has its infrastructure needs. Recent expansion of the operations team has allowed us to replace our existing service discovery solution with Consul, improving our ability to scale and manage an elastic cloud environment. At the same time,we saw this opportunity to migrate from EC2 Classic to VPC and take advantage of AWS's latest offerings. In this talk, we will discuss Consul, the problems it has solved, and adoption issues that surfaced along the way. In addition, we will also highlight our experiences with VPC, including setup, routing, access control, and migration with the extremely useful EC2 ClassicLink. PDF with presenter notes and links can be found here: http://bit.ly/1OH7HC0

Scalable networking in Apache CloudStackChiradeep Vittal

This document discusses networking in Apache CloudStack and the challenges of scaling cloud networking. It provides an overview of Apache CloudStack, including its flexibility to support various hypervisors, network topologies, and storage options. It then covers how CloudStack supports different networking modes, from traditional VLAN-based isolation to overlay networks. It also discusses how CloudStack orchestrates network resources and services to provide configurable network offerings to end users. Finally, it compares traditional server virtualization networking with the cloud approach and covers options like software defined networking.

VMUG St Louis - SDN in the Real WorldChris Wahl

This document provides an overview of Chris Wahl and his expertise in networking and virtualization. It summarizes his background as a VMware Certified Design Expert and Pluralsight author. The bulk of the document then summarizes key concepts regarding software-defined networking and how NSX implements networking, security and virtualization services in a virtualized environment in 3 sentences or less per section. It also briefly outlines example use cases and approaches to infrastructure as code and network automation using NSX.

Amazon Virtual Private Cloud - VPC 2AWS Riyadh User Group

The document outlines the agenda for a user group meeting on AWS VPC topics. The agenda includes reviewing default and custom VPCs, NAT instances and gateways, VPC peering, flow logs, endpoints, VPN connections, Direct Connect, limits and pricing, and exam tips. It also lists past topics such as storage, compute, databases, and networking services, as well as upcoming topics such as Lambda, cost optimization, and machine learning.

Intro to CloudStackJoe Brockmeier

This document provides an overview and introduction to CloudStack, an open source cloud computing platform. It outlines CloudStack's history and background, key features including hypervisor support, scalability, and high availability. It also describes CloudStack's architecture and components, deployment models, and how it provides infrastructure as a service capabilities. The document concludes with an announcement of CloudStack joining the Apache Software Foundation and being governed as an Apache project.

CloudStack Overviewsedukull

Windows Azure Virtual MachinesNeil Mackenzie

Introduction to CloudStack: How to Deploy and Manage Infrastructure-as-a-Serv...cloud-diva

What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlueShapeBlue

Giles’s talk provided a detailed overview of the latest advancements and upcoming features in Apache CloudStack. Giles explored the 4.19 release, highlighting significant updates such as VNF Appliances Support, KVM Import, and VMware to KVM migration. Looking ahead to version 4.20, the session outlined anticipated features and offer a quick sneak peek into the project roadmap. -- The CloudStack European User Group 2024 took place on September 19th in Frankfurt, Germany. This year's event was another successful User Group for the community. The event drew a strong audience, highlighting the growth of the CloudStack community in the region.

VMworld 2013: Virtualized Network Services Model with VMware NSX VMworld

This document summarizes a presentation about VMware's NSX virtualized networking solution. It introduces NSX Edge gateways which provide routing, firewalling, load balancing, and VPN services. It discusses how NSX addresses the needs of cloud computing through automation, standard hardware, and a single management plane. Example use cases are shown. Key features of the NSX Edge including scalable performance are outlined. The document also briefly discusses NSX operations and management tools, and its deployment on VMware vCloud Hybrid Service.

Intro to ServerlessCliff Chao-kuan Lu

Network Functions Virtualization and CloudStackChiradeep Vittal

PLNOG 17 - Tomasz Stachlewski - Infrastruktura sieciowa w chmurze AWSPROIDEA

Celem prezentacji jest przedstawienie sposobu tworzenia i zarządzania infrastrukturą sieciową w chmurze (AWS). Podczas prezentacji użytkownicy dowiedzą się z jakich komponentów składa się infrastruktura w chmurze, zapoznają się z tematyką VPC (Virtual Private Cloud), Security Group, Direct Connect, Avaibility Zone, Route53, Regions. Dodatkowo dowiedzą się jak należy projektować systemy aby były określane jako HA oraz w jaki sposób można tworzyć rozwiązania hybrydowe i połączyć chmurę z istniejącą infrastrukturą on-premise. Dodatkowo słuchacze zapoznają się ze sposobem zarządzania infrastrukturą sieciową jak kodem (tzw. IaC - Infrastructure as Code) – co pozwala w szybki sposób tworzyć i zarządzać całością infrastruktury sieciowej w chmurze.

Chapter 8. Cloud computing Overview for engineeringNhân Lê

Why and How CloudStack at weSystems - Stephan Bienek - weSystemsShapeBlue

Why? What? How? During the session, Stephan shares what made weSystems choose CloudStack as their main platform, which they base most of their managed services on. In addition, he talks about which services they provide based on CloudStack and how they integrated and use CloudStack today. ----------------------------------------- The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.

OpenStack and WindowsAlessandro Pilotti

"Turning Kubernetes into a full-fledged private cloud", Volodymyr TsapFwdays

Directions for CloudStack NetworkingChiradeep Vittal

The Future of SDN in CloudStack by Chiradeep Vittalbuildacloud

Hacking apache cloud stackNitin Mehta

Service Discovery: From Classic to VPCMark Corwin

Scalable networking in Apache CloudStackChiradeep Vittal

VMUG St Louis - SDN in the Real WorldChris Wahl

Amazon Virtual Private Cloud - VPC 2AWS Riyadh User Group

Intro to CloudStackJoe Brockmeier

CloudStack Overviewsedukull

Windows Azure Virtual MachinesNeil Mackenzie

Introduction to CloudStack: How to Deploy and Manage Infrastructure-as-a-Serv...cloud-diva

What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlueShapeBlue

VMworld 2013: Virtualized Network Services Model with VMware NSX VMworld

Intro to ServerlessCliff Chao-kuan Lu

Network Functions Virtualization and CloudStackChiradeep Vittal

PLNOG 17 - Tomasz Stachlewski - Infrastruktura sieciowa w chmurze AWSPROIDEA

Chapter 8. Cloud computing Overview for engineeringNhân Lê

Why and How CloudStack at weSystems - Stephan Bienek - weSystemsShapeBlue

More from confluent (20)

Webinar Think Right - Shift Left - 19-03-2025.pptxconfluent

Migration, backup and restore made easy using Kannikaconfluent

In this presentation, you’ll discover how easily you can migrate data from any Kafka-compatible event hub to Confluent using Kannika’s intuitive self-service interface. We’ll guide you through the process, showing how the same approach can be applied to define specific event data sets and effortlessly spin up secure environments for demos, testing, or other purposes. You’ll also learn how to back up event data in just a few steps by transferring compressed data to the cloud storage location of your choice. In addition, we’ll demonstrate how to restore filtered datasets of topics, ensuring quick recovery and maintaining business continuity when needed.

Five Things You Need to Know About Data Streaming in 2025confluent

Topics that Peter covers: Tapping into the Potential of Data Products: Data drives some of today's most important business use cases. Data products enable instant access to reliable and trustworthy data by eliminating the data mess created by point-to-point connections. The Need to Tap into 'Quick Thinking': The C-level has to reorient itself so it doesn't become the bottleneck to adaptability in a data-driven world. Nine in 10 (90%) business leaders say they must now react in real-time. Learn what you can do to provide executive access to real-time data to enable 'Quick Thinking.' Rise Above Data Hurdles: Discover how to enforce governance at data production. Reestablishing trustworthiness later is almost always harder, so investing in data tools that solve business problems rather than add to them is essential. Paradigm to Shift Left: Shift Left is a new paradigm for processing and governing data at any scale, complexity, and latency. Shift Left moves the processing and governance of data closer to the source, enabling organisations to build their data once, build it right and reuse it anywhere within moments of its creation. The Need for a Strategic View: The positive correlation between data streaming maturity and significant business returns underscores the importance of a long-term, strategic view of data streaming investments. It also highlights the value of advancing beyond initial, siloed use cases to a more integrated approach that leverages data streaming across the enterprise.

Data in Motion Tour Seoul 2024 - Keynoteconfluent

Data in Motion Tour Seoul 2024 - Roadmap Democonfluent

From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...confluent

In this presentation, we’ll demonstrate how Confluent and Lightstreamer come together to tackle the last-mile challenge of extending your Kafka architecture to web and mobile platforms. Learn how to effortlessly build real-time web applications within minutes, subscribing to Kafka topics directly from your web pages, with unmatched low latency and high scalability. Explore how Confluent's leading Kafka platform and Lightstreamer's intelligent proxy work seamlessly to bridge Kafka with the internet frontier, delivering data in real-time.

Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent

Data in Motion Tour 2024 Riyadh, Saudi Arabiaconfluent

Data streaming platforms are becoming increasingly important in today’s fast-paced world. From retail giants who need to monitor inventory levels to ensure stores never run out of items, to new-age, innovative banks who are building out-of-the-box banking solutions for traditional retail banks, data streaming platforms are at the centre, powering these workflows. Data streaming platforms connect all your applications, systems, and teams with a shared view of the most up-to-date, real-time data. From Gen AI, stream governance to stream processing - it’s these cutting edge developments that will be featured during the day.

Build a Real-Time Decision Support Application for Financial Market Traders w...confluent

Strumenti e Strategie di Stream Governance con Confluent Platformconfluent

Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeksconfluent

As businesses strive to stay at the forefront of innovation, the ability to quickly develop scalable Generative AI (GenAI) applications is essential. Join us for an exclusive webinar featuring MIA Platform, MongoDB, and Confluent, where you'll learn how to compose GenAI apps with real-time data integration in a fraction of the time. Discover how these three powerful platforms work together to ensure applications remain responsive, relevant, and adaptive to user preferences and contextual changes. Our experts will guide you through leveraging MIA Platform's microservices architecture and low-code development, MongoDB's flexibility, and Confluent's stream processing capabilities. Experience live demonstrations and practical insights that will transform your approach to AI-driven app development, enabling you to accelerate your development process from weeks to mere minutes. Don't miss this opportunity to keep your business at the cutting edge.

Building Real-Time Gen AI Applications with SingleStore and Confluentconfluent

Unlocking value with event-driven architecture by Confluentconfluent

Sfrutta il potere dello streaming di dati in tempo reale e dei microservizi basati su eventi per il futuro di Sky con Confluent e Kafka®. In questo tech talk esploreremo le potenzialità di Confluent e Apache Kafka® per rivoluzionare l'architettura aziendale e sbloccare nuove opportunità di business. Ne approfondiremo i concetti chiave, guidandoti nella creazione di applicazioni scalabili, resilienti e fruibili in tempo reale per lo streaming di dati. Scoprirai come costruire microservizi basati su eventi con Confluent, sfruttando i vantaggi di un'architettura moderna e reattiva. Il talk presenterà inoltre casi d'uso reali di Confluent e Kafka®, dimostrando come queste tecnologie possano ottimizzare i processi aziendali e generare valore concreto.

Il Data Streaming per un’AI real-time di nuova generazioneconfluent

Per costruire applicazioni di AI affidabili, sicure e governate occorre una base dati in tempo reale altrettanto solida. Ancor più quando ci troviamo a gestire ingenti flussi di dati in continuo movimento. Come arrivarci? Affidati a una vera piattaforma di data streaming che ti permetta di scalare e creare rapidamente applicazioni di AI in tempo reale partendo da dati affidabili. Scopri di più! Non perdere il nostro prossimo webinar durante il quale avremo l’occasione di: • Esplorare il paradigma della GenAI e di come questa nuova tecnnologia sta rimodellando il panorama aziendale, rispondendo alla necessità di offrire un contesto e soluzioni in tempo reale che soddisfino le esigenze della tua azienda. • Approfondire le incertezze del panorama dell'AI in evoluzione e l'importanza cruciale del data streaming e dell'elaborazione dati. • Vedere in dettaglio l'architettura in continua evoluzione e il ruolo chiave di Kafka e Confluent nelle applicazioni di AI. • Analizzare i vantaggi di una piattaforma di streaming dei dati come Confluent nel collegare l'eredità legacy e la GenAI, facilitando lo sviluppo e l’utilizzo di AI predittive e generative.

Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent

As businesses strive to remain at the cutting edge of innovation, the demand for scalable and up-to-date conversational AI solutions has become paramount. Generative AI (GenAI) chatbots that seamlessly integrate into our daily lives and adapt to the ever-evolving nuances of human interaction are crucial. Real-time data plays a pivotal role in ensuring the responsiveness and relevance of these chatbots, empowering them to stay abreast of the latest trends, user preferences, and contextual information.

Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent

Building API data products on top of your real-time data infrastructureconfluent

This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products. You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering. You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks. Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!

Speed Wins: From Kafka to APIs in Minutesconfluent

Evolving Data Governance for the Real-time Streaming and AI Eraconfluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent