Taking Your Database Beyond the Border of a Single Kubernetes Cluster

Oct 18, 20210 likes84 views

The document discusses expanding an Apache Cassandra database deployment from a single Kubernetes cluster across multiple clusters. It notes some challenges with networking, address collisions, and service discovery that must be addressed when deploying a database across cluster boundaries. Potential solutions discussed include using overlay networks, managed DNS, Istio, Cilium, and customizing Kubernetes DNS configurations to help route traffic and discover services between clusters. The presenters aim to demonstrate how to deploy a multi-cluster Cassandra database managed by the K8ssandra operator.

Taking your
database beyond
the border of a single
Kubernetes cluster
Ty Morton, Google
Christopher Bradford, DataStax
1

Who are we?
Christopher Bradford, DataStax
K8ssandra Project Lead
@bradfordcp
Ty Morton, Google
Customer Engineer
@tymorton
2

What are we trying to do?
Expand a single-region Apache
Cassandra cluster from one
Kubernetes environment to two.
Existing deployment:
● 1x region
● 3x zones
● 6x K8s workers
● 6x C* pods
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 3

What are we trying to do?
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 4

Seems simple...
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 5

Networking
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 8

What are we trying to do?
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 9

Address Collisions
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 10

Flatten It Out
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 11

It depends...
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 12

Overlay Networks
13
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax)

Service Discovery
14
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax)

DNS Stubs
Kubernetes has support for DNS stubs
us-west4.cluster.local:53 {
errors
cache 30
forward . 10.150.0.1
}
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 15

Managed DNS
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 16

Strong Alternatives
Cilium Istio
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 17

Added Beneﬁts
18
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax)

Deployment
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 19

References
CIDR Wikipedia
Istio Multicluster Guide
LinkerD Multicluster Guide
Google GKE Cloud DNS
Cilium Cluster Mesh
Cilium Cassandra Secure Access
Kubernetes Docs - Customize DNS
Yahoo! Japan Blog Post - Discusses
BGP and K8s for easy routing
DataStax Overlay Network Blog Post
K8ssandra Multicluster Operator
Taking your database beyond the border of a single Kubernetes cluster • Ty Morton (Google) & Christopher Bradford (DataStax) 21

Deploying and running applications globally, at scale, is becoming easier and faster with Kubernetes. From a single container Express app to microservice-powered service meshes, cloud-native applications are the norm. In this session, Christopher Bradford explores the aspects you’ll want to consider when taking the database of your application global including: Deployment topologies like hybrid and multi-cloud Infrastructure and configuration planning Resiliency and planning for failure Security and privacy Application connectivity After this session, you’ll be equipped with the bigger picture of what it takes to move your data operations onto Kubernetes in a globally distributed space, as part of a cloud-native stack.

Ceph scale testing with 10 Billion ObjectsKaran Singh

Beyond unit tests: Deployment and testing for Hadoop/Spark workflowsDataWorks Summit

As a Hadoop developer, do you want to quickly develop your Hadoop workflows? Do you want to test your workflows in a sandboxed environment similar to production? Do you want to write unit tests for your workflows and add assertions on top of it? In just a few years, the number of users writing Hadoop/Spark jobs at LinkedIn have grown from tens to hundreds and the number of jobs running every day has grown from hundreds to thousands. With the ever increasing number of users and jobs, it becomes crucial to reduce the development time for these jobs. It is also important to test these jobs thoroughly before they go to production. We’ve tried to address these issues by creating a testing framework for Hadoop/Spark jobs. The testing framework enables the users to run their jobs in an environment similar to the production environment and on the data which is sampled from the original data. The testing framework consists of a test deployment system, a data generation pipeline to generate the sampled data, a data management system to help users manage and search the sampled data and an assertion engine to validate the test output. In this talk, we will discuss the motivation behind the testing framework before deep diving into its design. We will further discuss how the testing framework is helping the Hadoop users at LinkedIn to be more productive.

Pachyderm: Building a Big Data Beast On KubernetesKubeAcademy

Pachyderm is a containerized data analytics solution that's completely deployed using Kubernetes. We take all the amazing tools and potential in the container ecosystem and unlock that power for massive-scale data processing. In this talk we'll show you how to leverage Docker, Kubernetes, and Pachyderm, to build incredibly robust and scalable data infrastructure. We'll start by discussing the key components of a modern data-drive company and how your infrastructure choices can have a massive impact on your product and scalability roadmap. We'll then dive into some architecture details to show how Kubernetes, Docker, and Pachyderm all work in tandem to create a cohesive data infrastructure stack. Finally, we will demonstrate some high-level use cases and powerful benefits you get from the architecture we've outlined. KubeCon schedule link: http://sched.co/4WWA

Data relay introduction to big data clustersChris Adkin

This document provides an overview of SQL Server 2019 Big Data Clusters, which enable hybrid SQL Server/Spark scale-out data platforms that run on Kubernetes. Big Data Clusters are available in public preview and will generally be available in the second half of 2019. They provide a true scale-out data platform for aggregating data from various sources, using data science tools with sensitive data on the same platform, and storing/querying large amounts of unstructured data with SQL Server tools.

Bruno Silva - eMedLab: Merging HPC and Cloud for Biomedical ResearchDanny Abukalam

This document discusses eMedLab, a project that merges HPC and cloud infrastructure for biomedical research. The key points are: - eMedLab provides a shared HPC and cloud environment across several UK biomedical institutions to enable data-driven medical research. - It uses an OpenStack cloud with an IBM-provided HPC cluster comprising over 100 servers and petabytes of storage. - Several pilot projects are underway utilizing the infrastructure for tasks like genome analysis, integrating clinical data, and collaborative cancer genomics. - Challenges include integrating support across institutions, ensuring security and performance when sharing storage, and addressing the differences between HPC and cloud computing models. - Future plans

Hyperconverged Cloud, Not just a toy anymore - Andrew Hatfield, Red HatOpenStack

Audience Level Intermediate Synopsis Hypercoverged Compute, Network and Storage is ready for production workloads – where it makes sense. Whether you’re a telecommunications carrier, service provider or enterprise; implementing Network Function Virtualisation (NFV), focusing on specific known workloads or simply a dev / test cloud – deploying a hypercoverged OpenStack cloud makes a lot of sense. Come along and discover which workloads fit a hyperconverged architecture, see examples and look into the very near future and learn how OpenStack is truly ready to serve your every need. Speaker Bio Andrew has over 20 years experience in the IT industry across APAC, specialising in Databases, Directory Systems, Groupware, Virtualisation and Storage for Enterprise and Government organisations. When not helping customers slash costs and increase agility by moving to the software-defined future, he’s enjoying the subtle tones of Islay Whisky and shredding pow pow on the world’s best snowboard resorts.

Bdc from bare metal to k8sChris Adkin

This document provides an overview of how to deploy a SQL Server 2019 Big Data Cluster on Kubernetes. It discusses setting up infrastructure with Ubuntu templates, installing Kubespray to manage the Kubernetes cluster lifecycle, and using azdata to deploy the Big Data Cluster. Key steps include creating an Ansible inventory, configuring storage with labels and profiles, and deploying the cluster. The document also offers tips on sizing, upgrades, and next steps like load balancing and monitoring.

Data weekender deploying prod grade sql 2019 big data clustersChris Adkin

Build a Complex, Realtime Data Management App with Postgres 14!Jonathan Katz

Congratulations: you've been selected to build an application that will manage reservations for rooms! On the surface, this sounds simple, but you are building a system for managing a high traffic reservation web page, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the website checking to see what availability each room has. Fortunately, PostgreSQL is prepared for this! And even better, we will be using Postgres 14 to make the problem even easier! We will explore the following PostgreSQL features: * Data types and their functionality, such as: * Data/Time types * Ranges / Multirnages Indexes such as: * GiST * Common Table Expressions and Recursion (though multiranges will make things easier!) * Set generating functions and LATERAL queries * Functions and the PL/PGSQL * Triggers * Logical decoding and streaming We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding. At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all users made possible by the innovation of PostgreSQL!

ELK: Moose-ively scaling your log systemAvleen Vig

This document discusses Etsy's journey with Elasticsearch, Logstash, and Kibana (ELK) over 3 years. It covers lessons learned in monitoring and scaling Logstash and Elasticsearch clusters. Key topics include sizing Elasticsearch and Logstash clusters based on resources like CPU, memory, disk I/O, and networking. Monitoring systems and metrics is also discussed. The document provides advice on optimizing Logstash performance by measuring baselines, managing garbage collection, and writing custom plugins when needed. Testing configuration changes is emphasized.

RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...InfluxData

Nowadays, every single modern application, system or solution does expose a RESTful API. On one hand, this is absolutely great and it has led to where we are today, having hundreds of other solutions or applications that can leverage these APIs, extend them, or even build on top of them. On the other hand, we have difficulty monitoring these new and modern systems, applications or solutions. In this session, we will learn how to query the data first using Swagger, when available, extract and parse the data that’s useful for us, store it in InfluxDB, and finally how to create beautiful and meaningful dashboards to have everything on a single pane of glass.

Storing Cassandra Metrics (Chris Lohfink, DataStax) | C* Summit 2016DataStax

Discussion about the evolution of metrics in Cassandra from 1.0 to 3.0, how the metric changes impact operational tooling, pros and cons for different metric representations, and how and why DataStax OpsCenter collects and stores metrics. Includes a deep dive on how DataStax OpsCenter represents and stores the different kinds of metrics to provide visibility beyond simple cluster averages both behind the scenes and in the rendering. About the Speaker Chris Lohfink Software Engineer, DataStax I am a Java, Python, and Clojure developer who has been using Cassandra in an application development and operational context for the last five years. The last nearly two years I have been working with the OpsCenter Monitoring team at DataStax to improve the accuracy and breadth of the visualization tooling available.

Ceph Object Storage Reference Architecture Performance and Sizing GuideKaran Singh

Building a Complex, Real-Time Data Management ApplicationJonathan Katz

Congratulations: you've been selected to build an application that will manage whether or not the rooms for PGConf.EU are being occupied by a session! On the surface, this sounds simple, but we will be managing the rooms of PGConf.EU, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the PGConf.EU website checking to see what availability each of the PGConf.EU rooms has. To do this, we will explore the following PGConf.EU features: * Data types and their functionality, such as: * Data/Time types * Ranges Indexes such as: * GiST * SP-Gist * Common Table Expressions and Recursion * Set generating functions and LATERAL queries * Functions and the PL/PGSQL * Triggers * Logical decoding and streaming We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding. At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all PGConf.EU attendees made possible by the innovation of PGConf.EU!

Seastar @ SF/BA C++UGAvi Kivity

Seastar is a C++ asynchronous programming framework that allows for multi-domain async programming across networking, storage I/O, and multi-core communications. It uses an event-driven model where each logical core runs a task scheduler independently. Logical cores communicate through queues. Seastar is applicable for workloads with high I/O to compute ratios, high concurrency needs, and distributed applications. It provides futures/promises abstractions and rich APIs for tasks like HTTP servers, RPC, and distributed databases.

ScyllaDB: NoSQL at Ludicrous SpeedJ On The Beach

ScyllaDB is a NoSQL database compatible with Apache Cassandra, distinguishing itself by supporting millions of operations per second, per node, with predictably low latency, on similar hardware. Achieving such speed requires a great deal of diligent, deliberate mechanical sympathy: ScyllaDB employs a totally asynchronous, share-nothing programming model, relies on its own memory allocators, and meticulously schedules all its IO requests. In this talk we will go over the low-level details of all the techniques involved - from a log-structured memory allocator to an advanced cache design -, covering how they are implemented and how they fully utilize the hardware resources they target.

Webinar: Does it Still Make Sense to do Big Data with Small Nodes?Julia Angell

In the world of Big Data, scaling out is the norm. However, many Big Data deployments are trapped in a sea of small box clusters. With the advent of scalable platforms like Scylla, node performance is no longer an issue and doubling the size of the nodes can double the available storage, memory, and processing power. So what stops people from going big in the Cloud Native world? Watch this webinar to learn the pros and cons of large nodes, and explore why people resist using big machines, including: - Is the cost of recovering from failures higher in larger nodes? - Does performance increase linearly as machines get bigger? - Does cluster performance suffer for the entire time of recovery from failures?

Data correlation using PySpark and HDFSJohn Conley

This document discusses using PySpark and HDFS to correlate different types of data at scale. It describes some challenges with correlating out-of-order and high-volume data. It then summarizes three approaches tried: using Redis, RDD joins, and writing bindings to HDFS. The current recommended approach reads relevant binding buckets from HDFS to correlate records in small windows, supporting different temporal models. Configuration and custom logic can be plugged in at various points in the correlation process. While scalable, further improvements in latency and throughput are still needed.

Stig Telfer - OpenStack and the Software-Defined SuperComputerDanny Abukalam

This document discusses using OpenStack to create a software-defined supercomputer. It describes how the author's project took OpenStack and "hacked it into shape" to better support HPC workloads by developing solutions for issues like power control, networking, and diskless compute nodes. The goal is to achieve the flexibility of cloud computing without the typical performance overhead of cloud platforms, and to combine it with the programming environment of high-performance computing.

CEPH DAY BERLIN - DISK HEALTH PREDICTION AND RESOURCE ALLOCATION FOR CEPH BY ...Ceph Community

Ceph is intelligent. However, users usually make resource request with no guarantee because of no visibility of underlying disk health, no idea of resource availability and no prediction of future demands. Now machine learning can make it happen. We'll present how machine learning technologies help predict Ceph OSD health, predictive impact on clusters and resolutions. We'll take Kubernetes working with Ceph as an example.

Cassandra @ Yahoo Japan (Satoshi Konno, Yahoo) | Cassandra Summit 2016DataStax

We have been offering many internet services and smart phone applications for over 20 years in Japan, and Cassandra has been used by our services since 2010. In this presentation, I will explain some issues and solutions about Cassandra, and our next generation infrastructure for Cassandra. About the Speaker Satoshi Konno Technical Manager, Yahoo Japan Corporation Satoshi Konno is a software engineer with 20 years of experience. He has worked in Yahoo Japan as a programmer for 10 years and in their NoSQL team for the past 4 years and he is currently in a computer science doctoral course studying distributed computing.

PostgreSQL on AWS: Tips & Tricks (and horror stories)Alexander Kukushkin

At a blistering pace and for a variety of reasons, companies are migrating their on-premise database infrastructures to cloud-based solutions - to save costs on hardware, tame the impact of disaster recovery, or even to improve security. Zalando is not an exception: more than two years ago we migrated our first production services to AWS. In addition to the fully managed database services like RDS and Aurora, Amazon offers a wide spectra of EC2 Instances with different types of performance and price. Without a lot of experience in running cloud databases it’s not easy to make the right choice, and as a result you will either have pure database performance or will overpay for over-provisioned resources. In this talk I will compare different ways of running PostgreSQL on AWS, explain why we decided to run most of our databases on EC2 Instances instead of RDS, how we chose EC2 Instance types and EBS Volumes, which AWS CloudWatch metrics MUST be monitored (and why), and what problems we hit plus how to avoid them.

Lessons Learned: Running InfluxDB Cloud and Other Cloud Services at Scale | T...InfluxData

Spark Pipelines in the Cloud with Alluxio with Gene PangSpark Summit

Organizations commonly use Apache Spark to gain actionable insight from their large amounts of data. Often, these analytics are in the form of data processing pipelines, where there are a series of processing stages, and each stage performs a particular function, and the output of one stage is the input of the next stage. There are several examples of pipelines, such as log processing, IoT pipelines, and machine learning. The common attribute among different pipelines is the sharing of data between stages. It is also common for Spark pipelines to process data stored in the public cloud, such as Amazon S3, Microsoft Azure Blob Storage, or Google Cloud Storage. The global availability and cost effectiveness of these public cloud storage services make them the preferred storage for data. However, running pipeline jobs while sharing data via cloud storage can be expensive in terms of increased network traffic, and slower data sharing and job completion times. Using Alluxio, a memory speed virtual distributed storage system, enables sharing data between different stages or jobs at memory speed. By reading and writing data in Alluxio, the data can stay in memory for the next stage of the pipeline, and this result in great performance gains. In this talk, we discuss how Alluxio can be deployed and used with a Spark data processing pipeline in the cloud. We show how pipeline stages can share data with Alluxio memory for improved performance benefits, and how Alluxio can improves completion times and reduces performance variability for Spark pipelines in the cloud.

Scylla Summit 2016: Analytics Show Time - Spark and Presto Powered by ScyllaScyllaDB

Ceph Object Storage Performance Secrets and Ceph Data Lake SolutionKaran Singh

Critical Attributes for a High-Performance, Low-Latency DatabaseScyllaDB

This document discusses the attributes of a high-performance, low-latency database like ScyllaDB. It begins with introductions and an overview of ScyllaDB. It then summarizes how hardware has evolved over 20 years with more cores, memory, and faster disks. ScyllaDB was redesigned from first principles to take advantage of modern hardware, using an asynchronous, shared-nothing architecture with one shard per core. This allows it to achieve significantly higher performance than Cassandra. The document shows benchmark results demonstrating ScyllaDB's lower latencies and ability to scale to higher throughput. It also discusses how ScyllaDB uses workload prioritization to manage different types of workloads.

Workshop on Google Cloud Data PlatformGoDataDriven

The document provides an agenda and information about a GoDataFest workshop on Google Cloud Platform for data. The agenda includes an introduction to GCP for data, a session on roles and tools on GCP for different data roles, and a session where participants will build projects on GCP in mixed workgroups. It outlines the goals and tools used by different roles like data engineer, analytics engineer, and Looker user. It also provides information on Google Cloud technologies like BigQuery, Dataform, Looker, and how they fit into the modern data lifecycle and platform. Participants are then divided into mixed workgroups based on their preferred role and given insights to explore in their projects.

Building a Multi-Region Cluster at Target (Aaron Ploetz, Target) | Cassandra ...DataStax

Lessons learned from a year spent building a Cassandra cluster over multiple regions, data centers, and providers. Will discuss our successes and learnings on replication, operations, and application development. About the Speaker Aaron Ploetz Lead Technical Architect, Target Aaron is a Lead Technical Architect for Target, where he coaches development teams on modeling and building applications for Cassandra. He is active in the Cassandra tags on StackOverflow, and has also contributed patches to cqlsh. Aaron holds a B.S. in Management/Computer Systems from the University of Wisconsin-Whitewater, a M.S. in Software Engineering and Database Technologies from Regis University, and is a 2x DataStax MVP for Apache Cassandra.

More Related Content

What's hot (20)