Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021

Jun 22, 2021Download as PPTX, PDF0 likes213 views

Apache Pulsar is used for various streaming use cases. There is a strong requirement for storing checkpoints while processing the stream in Pulsar Functions, so that in case of any interruption stream processing engine could go back to the last checkpoint. Pulsar uses Zookeeper not only for leader elections or service discovery like critical use cases but also for storing various metadata which puts unnecessary load on zookeeper which hampers mission critical use of Zookeeper. A durable key value store based off of the Apache Pulsar ecosystem addresses the above mentioned use cases nicely. This talk focuses on taking existing Apache Bookkeeper Table Service/State store implementation and taking it to production. Furthermore this talk also touches upon contributing all the features, bug fixes, tools and other improvements back to open source.

Pulsar Virtual Summit North America 2021
Apache BookKeeper State Store:
A Durable Key-Value Store

Pulsar Virtual Summit North America 2021
Prashant Kumar
Principal Software Engineer @ Splunk
● Principal Software Developer at Splunk.
● Ex Yahoo, Verizon Media.
● Prior Experience, Key member and
contributor to Sherpa, the geographically
replicated multi tenant key value store @
Yahoo.

Pulsar Virtual Summit North America 2021
Agenda
I. Introduction to Apache Bookkeeper Statestore.
II. Why yet another KV store?
III. How does it fit into Pulsar ecosystem?
IV. Intended use case.
V. Brief architecture.
VI. Current state and production worthiness.
VII. Product roadmap and future work.

Pulsar Virtual Summit North America 2021
Apache Bookkeeper Statestore
● It’s a Key-Value store.
● It’s durable.
● It’s locally replicated.
● It’s eventually consistent.
● It’s fault tolerant.
● It’s cloud native and k8s based deployment.

Pulsar Virtual Summit North America 2021
Argh!. Yet another KV store?

Pulsar Virtual Summit North America 2021
Integral part of Apache Pulsar ecosystem
● Uses same Zookeeper deployment that Pulsar uses.
● Uses same Bookkeeper deployment that Pulsar uses.
● Uses same infrastructure for metrics, dashboards etc as
Bookkeeper
● Part of bookkeeper/stream code base.
● Existing client side integration in Apache Pulsar function
service.

Pulsar Virtual Summit North America 2021
Primary use cases
● Store and access function state and checkpoints
● A secondary metadata store for Apache Pulsar, away from
Zookeeper
● Other various KV store use cases

Pulsar Virtual Summit North America 2021
Data model

Pulsar Virtual Summit North America 2021
High level serving architecture (Bird view)

Pulsar Virtual Summit North America 2021
High level Datastore architecture

Pulsar Virtual Summit North America 2021
Benchmarking
● Benchmarking with YCSB
● Setup
○ YCSB Thread count = 40
○ # k8s pods = 3
○ cpuRequest = 8
○ cpuLimit = 16
○ memoryRequest = 24Gi
○ memoryLimit = 24Gi
● Read output
○ Throughput = 22557.7 Ops/S
○ Average latency = 1.699 ms
○ 99%tile latency = 5.323 ms
● Write output
○ Throughput = 15256.16 Ops/S
○ Average latency = 8.820 ms
○ 99%tile = 27.071 ms

Pulsar Virtual Summit North America 2021
Production readiness
● It has already been in production for last few
months
● It’s a k8s based deployment
● Sustained production traffic
○ Read throughput 240 Ops/S
○ Write throughput 90 Ops/S

Pulsar Virtual Summit North America 2021
Product roadmap
● Contribute internal changes back to open source
● Storage hardening
○ Pick up the data litter actively and reactively
○ Clear obsolete transaction log
● Operability
○ Improved and granular monitoring and alerting.
● Availability
○ Implementation of replica.
○ Serving read traffic from a replica.
○ Elevation of replica to be primary when primary fails
● Scaling and load balancing
○ Splitting a shard
○ Moving shards across cluster

Pulsar Virtual Summit North America 2021
References
● Statestore Repo:
https://github.com/apache/bookkeeper/tree/master/stream
● Distributedlog Repo:
https://github.com/apache/bookkeeper/tree/master/stream/distri
butedlog
● Pulsar Function - Statestore integration:
https://github.com/apache/pulsar/blob/master/pulsar-
functions/worker/src/main/java/org/apache/pulsar/functions/wor
ker/PulsarWorkerService.java#L420

In order to leverage the best performance characters of your data or stream backend, it is important to understand the nitty gritty details of how your backend store and compute works, how data is stored, how is it indexed and how the read path is. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this slide deck, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! The focus will be more on storage backend so as to not keep this tailored to pulsar specifically but to be able to apply it different data stores or streams.

Kafka for DBAsGwen (Chen) Shapira

This document discusses Apache Kafka and how it can be used by Oracle DBAs. It begins by explaining how Kafka builds upon the concept of a database redo log by providing a distributed commit log service. It then discusses how Kafka is a publish-subscribe messaging system and can be used to log transactions from any database, application logs, metrics and other system events. Finally, it discusses how schemas are important for Kafka since it only stores messages as bytes, and how Avro can be used to define and evolve schemas for Kafka messages.

Flink vs. SparkSlim Baltagi

Kafka Reliability - When it absolutely, positively has to be thereGwen (Chen) Shapira

Kafka provides reliability guarantees through replication and configuration settings. It replicates data across multiple brokers to protect against failures. Producers can ensure data is committed to all in-sync replicas through configuration settings like request.required.acks. Consumers maintain offsets and can commit after processing to prevent data loss. Monitoring is also important to detect any potential issues or data loss in the Kafka system.

Deep Dive into Apache Kafkaconfluent

In the last few years, Apache Kafka has been used extensively in enterprises for real-time data collecting, delivering, and processing. In this presentation, Jun Rao, Co-founder, Confluent, gives a deep dive on some of the key internals that help make Kafka popular. - Companies like LinkedIn are now sending more than 1 trillion messages per day to Kafka. Learn about the underlying design in Kafka that leads to such high throughput. - Many companies (e.g., financial institutions) are now storing mission critical data in Kafka. Learn how Kafka supports high availability and durability through its built-in replication mechanism. - One common use case of Kafka is for propagating updatable database records. Learn how a unique feature called compaction in Apache Kafka is designed to solve this kind of problem more naturally.

Pulsar Watermarking - Pulsar Virtual Summit Europe 2021StreamNative

The Rise Of Event Streaming – Why Apache Kafka Changes EverythingKai Wähner

Business digitalization trends like microservices, the Internet of Things or Machine Learning are driving the need to process events at a whole new scale, speed and efficiency. Traditional solutions like ETL/data integration or messaging are not build to serve these needs. Today, the open source project Apache Kafka® is being used by thousands of companies including over 60% of the Fortune 100 to power and innovate their businesses by focusing their data strategies around event-driven architectures leveraging event streaming.We will discuss the market and technology changes that have given rise to Kafka and to Event Streaming, and we will introduce the audience to the key aspects of building an Event streaming platform with Kafka. Examples of productive use cases from the automotive, manufacturing and transportation sector will showcase the power of event streaming.

Kafka Tutorial - introduction to the Kafka streaming platformJean-Paul Azar

How Pulsar Stores Your Data - Pulsar Summit NA 2021StreamNative

In order to leverage the best performance characters of your stream backend, it is important to understand the nitty gritty details of how pulsar stores your data. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this talk, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! This session will empower you with the right background to map your data right with pulsar.

Kafkaを使ったマイクロサービス基盤 part2 ＋運用して起きたトラブル集matsu_chara

AWS Black Belt Tech シリーズ 2015 - AWS Data PipelineAmazon Web Services Japan

AWS Black Belt Tech Webinar 2015 Amazon Kinesis 次回のWebinarは、下記よりご確認ください。 http://aws.amazon.com/jp/about-aws/events/#webinar ★AWS Black Belt Tech Webinarとは AWSのソリューションアーキテクト、TechメンバがAWSのプロダクト/ソリューションを深堀りして解説し、参加されている皆さまからの質問にお答えする無料のWebinar(Webセミナー）です。毎週水曜日(祝日などを除く)、日本時間の18:00から約1時間にわたってお送りしています。

PGConf.ASIA 2017 Logical Replication Internals (English)Noriyoshi Shinoda

Streaming platform Kafka in SK planetByeongsu Kang

Thrift vs Protocol Buffers vs Avro - Biased ComparisonIgor Anishchenko

Igor Anishchenko Odessa Java TechTalks Lohika - May, 2012 Let's take a step back and compare data serialization formats, of which there are plenty. What are the key differences between Apache Thrift, Google Protocol Buffers and Apache Avro. Which is "The Best"? Truth of the matter is, they are all very good and each has its own strong points. Hence, the answer is as much of a personal choice, as well as understanding of the historical context for each, and correctly identifying your own, individual requirements.

kafkaAmikam Snir

Kafka is an open-source distributed commit log service that provides high-throughput messaging functionality. It is designed to handle large volumes of data and different use cases like online and offline processing more efficiently than alternatives like RabbitMQ. Kafka works by partitioning topics into segments spread across clusters of machines, and replicates across these partitions for fault tolerance. It can be used as a central data hub or pipeline for collecting, transforming, and streaming data between systems and applications.

超実践 Cloud Spanner 設計講座Samir Hammoudi

AWS Lambdaを紐解くKeisuke Nishitani

Kafka 101Clement Demonchy

Building flexible ETL pipelines with Apache Camel on QuarkusIvelin Yanev

This document discusses building flexible ETL pipelines with Apache Camel on Quarkus. It begins with an overview of what ETL is and the extract, transform, load process. It then discusses what Apache Camel is and how it is an open source integration framework that allows defining routing and mediation rules. The document introduces Camel K and Camel Quarkus, noting that Camel Quarkus brings Camel's integration capabilities to the Quarkus runtime. It argues that Apache Camel and Quarkus is a good combination for efficient ETL due to Camel's easy learning curve and extensibility and Quarkus' benefits like low memory usage and fast startup times. The document concludes with a demo

[2D4]Python에서의 동시성_병렬성NAVER D2

MySQL Binlog Events でストリーム処理してみた #MySQLUC15Yahoo!デベロッパーネットワーク

探索 API 開發的挑戰與解決之道 | .NET Conf 2023 TaiwanAlan Tsai

相信大家對於為什麼要把服務 API 化的原因都清楚了，但是該怎麼開始呢？如果要走 API Design First 那怎麼可以產生出好持續維護的 OpenAPI (Swagger) 文件作為溝通的 Contract 呢？API 開發完上線之後怎麼能夠確保安全，以及出錯如何追蹤問題呢？如果只是單純的 DB 異動的 CRUD 需要建立後端 API 嗎？隨著 API 數量變多怎麼收攏所有有提供的 API 呢？在這場，我們來看一下針對一些 API First 遇到的問題，微軟提供了那些解決方案給我們。

Fluentdのお勧めシステム構成パターンKentaro Yoshida

Kafka replication apachecon_2013Jun Rao

The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.

Common issues with Apache Kafka® Producerconfluent

Badai Aqrandista, Confluent, Senior Technical Support Engineer This session will be about a common issue in the Kafka Producer: producer batch expiry. We will be discussing the Kafka Producer internals, its common causes, such as a slow network or small batching, and how to overcome them. We will also be sharing some examples along the way! https://www.meetup.com/apache-kafka-sydney/events/279651982/

Helidon 概要オラクルエンジニア通信

Spark with Delta LakeKnoldus Inc.

Delta Lake is an open source storage layer that sits on top of data lakes and brings ACID transactions and reliability to Apache Spark. It addresses challenges with data lakes like lack of schema enforcement and transactions. Delta Lake provides features like ACID transactions, scalable metadata handling, schema enforcement and evolution, time travel/data versioning, and unified batch and streaming processing. Delta Lake stores data in Apache Parquet format and uses a transaction log to track changes and ensure consistency even for large datasets. It allows for updates, deletes, and merges while enforcing schemas during writes.

JupyterHub: Learning at ScaleCarol Willing

Manage Pulsar Cluster Lifecycles with Kubernetes Operators - Pulsar Summit NA...StreamNative

This document discusses how Pulsar operators can be used to automate lifecycle management of Pulsar clusters on Kubernetes. It describes how operators use custom resource definitions and controllers to reconcile the actual cluster state with the desired state. Specific examples are provided for how operators can perform controlled cluster upgrades, scale bookies, and clean up after cluster deletion. The integration of operators with Helm charts is also covered.

Benchmarking for postgresql workloads in kubernetesDoKC

ABSTRACT OF THE TALK 6 months have passed since our last DoK webinar about benchmarking PostgreSQL workloads in a Kubernetes environment. In the meantime, many things have happened at EDB, and we’re happy to share what we’ve learned in this timeframe. We’ll use cnp-bench and cnp-sandbox to help us describe some of the challenges we might face when running PostgreSQL workloads, how to spot them, and what actions to take to make your databases healthier and more longeve. cnp-bench is a collection of Helm charts that help run storage and database benchmarks, using popular open source tools like fio, pgbench, and HammerDB. cnp-sandbox is a Helm chart that sets up a Prometheus/Grafana stack, including basic metrics and dashboards for Cloud Native PostgreSQL, the Kubernetes operator developed by EDB. Both cnp-sandbox and cnp-bench are open source and recommended for development, testing, and pre-production environments only. BIO A long time open-source programmer and entrepreneur, Gabriele has a degree in Statistics from the University of Florence. After having consistently contributed to the growth of 2ndQuadrant and its members through nurturing a lean and devops culture, he is now leading the Cloud Native initiative at EDB. Gabriele lives in Prato, a small but vibrant city located in the northern part of Tuscany, Italy - famous for having hosted the first European PostgreSQL conferences. His second home is Melbourne, Australia, where he studied at Monash University and worked in the ICT sector. He loves playing the Blues with his Fender Stratocaster, but his major passions are called Elisabeth and Charlotte! KEY TAKE-AWAYS FROM THE TALK - A methodology for benchmarking a PostgreSQL database in Kubernetes - Open source set of tools for benchmarking a PostgreSQL database in Kubernetes - Reasons why benchmarking both the storage and the database is important https://github.com/EnterpriseDB/cnp-sandbox https://github.com/EnterpriseDB/cnp-bench

More Related Content

What's hot (20)

How Pulsar Stores Your Data - Pulsar Summit NA 2021StreamNative

Kafkaを使ったマイクロサービス基盤 part2 ＋運用して起きたトラブル集matsu_chara

AWS Black Belt Tech シリーズ 2015 - AWS Data PipelineAmazon Web Services Japan

PGConf.ASIA 2017 Logical Replication Internals (English)Noriyoshi Shinoda

Streaming platform Kafka in SK planetByeongsu Kang

Thrift vs Protocol Buffers vs Avro - Biased ComparisonIgor Anishchenko

kafkaAmikam Snir

超実践 Cloud Spanner 設計講座Samir Hammoudi

AWS Lambdaを紐解くKeisuke Nishitani

Kafka 101Clement Demonchy

Building flexible ETL pipelines with Apache Camel on QuarkusIvelin Yanev

[2D4]Python에서의 동시성_병렬성NAVER D2

MySQL Binlog Events でストリーム処理してみた #MySQLUC15Yahoo!デベロッパーネットワーク

探索 API 開發的挑戰與解決之道 | .NET Conf 2023 TaiwanAlan Tsai

Fluentdのお勧めシステム構成パターンKentaro Yoshida

Kafka replication apachecon_2013Jun Rao

Common issues with Apache Kafka® Producerconfluent

Helidon 概要オラクルエンジニア通信

Spark with Delta LakeKnoldus Inc.

JupyterHub: Learning at ScaleCarol Willing

How Pulsar Stores Your Data - Pulsar Summit NA 2021StreamNative

Kafkaを使ったマイクロサービス基盤 part2 ＋運用して起きたトラブル集matsu_chara

AWS Black Belt Tech シリーズ 2015 - AWS Data PipelineAmazon Web Services Japan

PGConf.ASIA 2017 Logical Replication Internals (English)Noriyoshi Shinoda

Streaming platform Kafka in SK planetByeongsu Kang

Thrift vs Protocol Buffers vs Avro - Biased ComparisonIgor Anishchenko

kafkaAmikam Snir

超実践 Cloud Spanner 設計講座Samir Hammoudi

AWS Lambdaを紐解くKeisuke Nishitani

Kafka 101Clement Demonchy

Building flexible ETL pipelines with Apache Camel on QuarkusIvelin Yanev

[2D4]Python에서의 동시성_병렬성NAVER D2

MySQL Binlog Events でストリーム処理してみた #MySQLUC15Yahoo!デベロッパーネットワーク

探索 API 開發的挑戰與解決之道 | .NET Conf 2023 TaiwanAlan Tsai

Fluentdのお勧めシステム構成パターンKentaro Yoshida

Kafka replication apachecon_2013Jun Rao

Common issues with Apache Kafka® Producerconfluent

Helidon 概要オラクルエンジニア通信

Spark with Delta LakeKnoldus Inc.

JupyterHub: Learning at ScaleCarol Willing

Similar to Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021 (20)

Manage Pulsar Cluster Lifecycles with Kubernetes Operators - Pulsar Summit NA...StreamNative

Benchmarking for postgresql workloads in kubernetesDoKC

Scylla Summit 2022: New AWS Instances Perfect for ScyllaDBScyllaDB

In this talk AWS’ Ken Krupa, Head of Specialized Solutions Architecture, will describe the architecture and capabilities of two new AWS EC2 instance types perfect for data-intensive storage and IO-heavy workloads like ScyllaDB: the Intel-based I4i and the Graviton2-based I4g series. The Intel Xeon Ice Lake-based I4i series provides unparalleled raw horsepower for your most demanding workloads. Meanwhile, the Graviton2-powered I4g instances provide lower cost per storage on a power-efficient platform to deploy your cloud-native applications. Ken will also describe the AWS Nitro SSD, a new form of high-speed NVMe storage with a Flash Translation Layer built with Nitro controllers, which powers both of these instance families. ScyllaDB VP of Product Tzach Livyatan will then share benchmarking results showing how ScyllaDB behaves under load on these two instance types, providing maximum system utility and efficiency. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Infinitic: Building a Workflow Engine on Top of Pulsar - Pulsar Summit NA 2021StreamNative

Infinitic is a free-to-use, source-available event-driven workflow engine built on top of Pulsar. Implementing a "workflow as code" pattern, Infinitic lets you write your distributed applications as if you were coding for a single infallible machine. It proposes a fault-tolerant stateful programming model that removes most of the complexities of building and orchestrating a distributed system. Infinitic makes intensive use of Pulsar features: functions, transactions, schema registry, subscription types, sinks, connectors. During this talk, I will showcase Infinitic capabilities and explain its architecture based on Pulsar.

Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community

The document summarizes a presentation given by representatives from various companies on optimizing Ceph for high-performance solid state drives. It discusses testing a real workload on a Ceph cluster with 50 SSD nodes that achieved over 280,000 read and write IOPS. Areas for further optimization were identified, such as reducing latency spikes and improving single-threaded performance. Various companies then described their contributions to Ceph performance, such as Intel providing hardware for testing and Samsung discussing SSD interface improvements.

In-memory No SQL- GIDS2014Hazelcast

This document provides an overview of Hazelcast, an open source in-memory data grid platform. It discusses why organizations are adopting NoSQL solutions like Hazelcast to overcome impedance mismatch between objects and relational databases. It provides details on Hazelcast's features, uses cases, customers, and roadmap. Hazelcast provides a distributed in-memory map for fast data access, clustering for high availability, and can be used for caching, session replication, and more.

Critical Attributes for a High-Performance, Low-Latency DatabaseScyllaDB

This document discusses the attributes of a high-performance, low-latency database like ScyllaDB. It begins with introductions and an overview of ScyllaDB. It then summarizes how hardware has evolved over 20 years with more cores, memory, and faster disks. ScyllaDB was redesigned from first principles to take advantage of modern hardware, using an asynchronous, shared-nothing architecture with one shard per core. This allows it to achieve significantly higher performance than Cassandra. The document shows benchmark results demonstrating ScyllaDB's lower latencies and ability to scale to higher throughput. It also discusses how ScyllaDB uses workload prioritization to manage different types of workloads.

How Pulsar Enables Netdata to Offer Unlimited Infrastructure Monitoring for F...StreamNative

The Netdata Agent is free, open source single-node monitoring software. Netdata Cloud is a free, closed source, software-as-a-service that brings together metadata from endpoints running the Netdata Agent, giving a complete view of the health and performance of an infrastructure. All the metrics remain on the Netdata Agent, making Netdata Cloud the focal point of a distributed, infinitely scalable, low cost solution.  The heart of Netdata Cloud is Pulsar. Almost every message coming from and going to the open source agents passes through Pulsar. Pulsar's infinite number of topics has given us the flexibility we needed and in some cases, every single Netdata Agent has its own unique Pulsar topic.  A single message from an agent or from a service that processes a front end request can trigger several other Pulsar messages, as we also use Pulsar for communication between microservices (using a CQRS pattern with shared subscriptions for scalability). The reliable persistence of messages has allowed us to replay old events to rebuild old and build new materialized views and debug specific production issues. It's also what will enable us to implement an event sourcing pattern, for a new set of features we want to introduce shortly.  We have had a few issues with a specific client and our shared subscriptions that we're working on resolving, but overall Pulsar has proven to be one of the most reliable parts of our infrastructure and we decided to proceed with a managed services agreement.

Oracle big data appliance and solutionssolarisyougood

The document summarizes Oracle's Big Data Appliance and solutions. It discusses the Big Data Appliance hardware which includes 18 servers with 48GB memory, 12 Intel cores, and 24TB storage per node. The software includes Oracle Linux, Apache Hadoop, Oracle NoSQL Database, Oracle Data Integrator, and Oracle Loader for Hadoop. Oracle Loader for Hadoop can be used to load data from Hadoop into Oracle Database in online or offline mode. The Big Data Appliance provides an optimized platform for storing and analyzing large amounts of data and is integrated with Oracle Exadata.

The architecture of SkySQLMariaDB plc

SkySQL implements a groundbreaking, state-of-the-art architecture based on Kubernetes and ServiceNow, and with a strong emphasis on cloud security – using compartmentalization and indirect access to secure and protect customer databases. In this session, we’ll walk through the architecture of SkySQL and discuss how MariaDB leverages an advanced Kubernetes operator and powerful ServiceNow configuration/workflow management to deploy and manage databases on cloud infrastructure.

Architectural caching patterns for kubernetesRafał Leszko

Self-hosting Kafka at Scale: Netflix's Journey & ChallengesNick Mahilani

Function Mesh: Complex Streaming Jobs Made Simple - Pulsar Summit NA 2021StreamNative

Pulsar Function is a succinct computing abstraction Apache Pulsar provides users to express simple ETL and streaming tasks. The simplicity comes in two folds: Simple Interface and Simple Deployment. As it has been adopted, we realized that the native support of organizing multiple functions into integrity will be very beneficial. With such support, people can express and manage multi-stage jobs easily. In addition, this support also provides the possibility of higher-level abstraction DSL to further simplify the job composition. We call this new feature -- Function Mesh. This talk aims to provide a thorough walkthrough of this new Function Mesh Feature, including its design, implementation, use cases, and examples, to help people seeking simple streaming solutions understand this newly created powerful tool in Apache Pulsar.

Learn from HomeAway Hadoop Development and Operations Best PracticesDriven Inc.

[DevConf.US 2019]Quarkus Brings Serverless to Java DevelopersDaniel Oh

This document discusses how Quarkus brings serverless capabilities to Java developers. It begins with an introduction of the speaker and outlines the developer journey from monolith applications to microservices and serverless functions. It then discusses challenges with existing serverless offerings and how Kubernetes and Knative address these. Quarkus is presented as a solution for serverless Java by providing benefits like developer productivity, fast and lightweight applications, and unifying imperative and reactive programming. The talk concludes by highlighting some of Quarkus' supported frameworks and providing additional resources.

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Nitin Kumar

Ohio Devfest - Visual Analysis with GCPWesley Workman

The document discusses visual analysis using Google Cloud Platform. It provides an overview of Google Cloud services including App Engine, Container Engine, Compute Engine, Datastore, Cloud SQL, and Spanner for building and deploying applications at scale. It also mentions Machine Learning services for tasks like natural language processing, image analysis, and translation. The presentation then demonstrates a photo sharing application built on these services and discusses ways to enhance it further using App Engine Flex, video processing, document parsing, BigQuery, and analytics. It concludes by inviting Python developers interested in visual analysis to discuss job opportunities.

YARNAlex Moundalexis

SVC / Storwize analysis cost effective storage planning (use case)Michael Pirker

This document discusses using BVQ software to perform a cost-effective storage planning analysis for migrating a 40TB mail system from 10k RPM disks to lower-cost 7.2k RPM disks. By analyzing I/O workload characteristics including read/write ratios and cache hit rates, the BVQ software determines that 48 7.2k disks would provide the required 2500 IOPS, reducing hardware costs. Additional ideas are provided to potentially reduce the required disk count further, such as challenging backup duration assumptions.

Retour d'expérience d'un environnement base de données multitenantSwiss Data Forum Swiss Data Forum

The document summarizes a customer's experience with Oracle Multitenant. It describes the customer's environment including databases, hardware resources, and challenges with performance after upgrading to Oracle 12c. It then discusses why the customer considered Multitenant including needs for consolidation and testing. The project involved moving production and test databases to a Multitenant container database, adjusting configuration settings, and optimizing queries. The results were improved performance and ability to scale resources. New features in Oracle 12.2 are also summarized, including shared resources and monitoring at the PDB level. Human: Thank you for the summary. Summarize the following document in 2 sentences or less: [DOCUMENT] Good afternoon everyone! Thank you for

Manage Pulsar Cluster Lifecycles with Kubernetes Operators - Pulsar Summit NA...StreamNative

Benchmarking for postgresql workloads in kubernetesDoKC

Scylla Summit 2022: New AWS Instances Perfect for ScyllaDBScyllaDB

Infinitic: Building a Workflow Engine on Top of Pulsar - Pulsar Summit NA 2021StreamNative

Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community

In-memory No SQL- GIDS2014Hazelcast

Critical Attributes for a High-Performance, Low-Latency DatabaseScyllaDB

How Pulsar Enables Netdata to Offer Unlimited Infrastructure Monitoring for F...StreamNative

Oracle big data appliance and solutionssolarisyougood

The architecture of SkySQLMariaDB plc

Architectural caching patterns for kubernetesRafał Leszko

Self-hosting Kafka at Scale: Netflix's Journey & ChallengesNick Mahilani

Function Mesh: Complex Streaming Jobs Made Simple - Pulsar Summit NA 2021StreamNative

Learn from HomeAway Hadoop Development and Operations Best PracticesDriven Inc.

[DevConf.US 2019]Quarkus Brings Serverless to Java DevelopersDaniel Oh

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...Nitin Kumar

Ohio Devfest - Visual Analysis with GCPWesley Workman

YARNAlex Moundalexis

SVC / Storwize analysis cost effective storage planning (use case)Michael Pirker

Retour d'expérience d'un environnement base de données multitenantSwiss Data Forum Swiss Data Forum

More from StreamNative (20)

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022StreamNative

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...StreamNative

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...StreamNative

The document discusses Toast's adoption and use of Apache Pulsar for asynchronous messaging in their microservices architecture. It describes how they built a "Pulsar Toggle" leveraging Envoy proxy to enable blue/green deployments of Pulsar consumers. The Pulsar Toggle allows consumers to be paused and resumed based on their status in the Envoy control plane, improving the reliability and usability of deploying changes to Pulsar-based services. Toast has seen increased adoption of Pulsar and benefits from its stability and scalability.

Distributed Database Design Decisions to Support High Performance Event Strea...StreamNative

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022StreamNative

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022StreamNative

This document summarizes Matteo Merli's talk on moving Apache Pulsar to a ZooKeeper-less metadata model. It discusses how Pulsar currently uses ZooKeeper for metadata storage but faces scalability issues. The talk outlines PIP-45, a plan to introduce a pluggable metadata backend into Pulsar to replace the direct ZooKeeper usage. This would allow alternative storage options like Etcd and improve scalability. It also discusses successes already achieved in Pulsar 2.10 by abstracting the metadata access and future goals around scaling to support millions of topics.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...StreamNative

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...StreamNative

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022StreamNative

This document summarizes the message redelivery process in Apache Pulsar. It discusses how messages are redelivered when producing or consuming messages. When producing, messages are redelivered if the broker does not acknowledge receipt in a timely manner. When consuming, messages are redelivered under three circumstances: if the acknowledgment times out, if messages are negatively acknowledged, or if delivery is delayed. The document provides details on the commands and objects involved in establishing connections, publishing, consuming, acknowledging, and redelivering messages between Pulsar clients and brokers.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...StreamNative

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Understanding Broker Load Balancing - Pulsar Summit SF 2022StreamNative

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...StreamNative

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022StreamNative

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Event-Driven Applications Done Right - Pulsar Summit SF 2022StreamNative

This document contains the agenda for a Pulsar Summit keynote on event-driven applications. The keynote will feature talks from Sijie Guo, Co-Founder and CEO of StreamNative, and Matteo Merli, CTO of StreamNative. Guo will discuss the growth of the Pulsar community and platform. Merli will cover the evolution of event-driven applications and the five fundamentals of modern event-driven architecture: data abstraction, API, primitives, processing semantics, and tools. The keynote aims to explain how Pulsar solves challenges in building complex event-driven applications.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022StreamNative

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022StreamNative

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022StreamNative

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Welcome and Opening Remarks - Pulsar Summit SF 2022StreamNative

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...StreamNative

Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data. Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic. Takeaways: 1) Get a general idea about what is a vector database and its real-world use cases. 2) Understand the major design principles of Milvus 2.0. 3) Learn how to build a complex system with the help of a modern log system like Pulsar.

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.