How Safe is Asynchronous Master-Master Setup?

Feb 4, 20200 likes538 views

This document discusses the risks of using asynchronous master-master replication for MySQL databases and provides strategies for setting it up safely. It explains that having two nodes actively accepting writes can lead to conflicts like duplicate key errors. It recommends dividing writes across nodes by database, table, or row to avoid conflicts. The document also discusses using synchronous replication tools like Galera to ensure consistency across nodes at the cost of reduced performance.

How Safe is Asynchronous Master-Master
February, 01, 2020
Sveta Smirnova

• MySQL Support engineer
• Author of
• MySQL Troubleshooting
• JSON UDF functions
• FILTER clause for MySQL
• Speaker
• Percona Live, OOW, Fosdem,
DevConf, HighLoad...
Sveta Smirnova
2

•What Happens with Two Active Nodes?
•Why to Write to Multiple Nodes?
•Galera, PXC, InnoDB Cluster
•How to Setup Safe Asynchronous Master-Master
Table of Contents
3

Master Slave
← Initiates
Asynchronous Built-In Replication
4

Master Slave
← Initiates
← Requests a packet
Asynchronous Built-In Replication
4

Master
Sends the packet →
Slave
← Initiates
← Requests a packet
Asynchronous Built-In Replication
4

Node1
INSERT SET id=42
Node2
INSERT SET id=42
INSERT
6

Node1
INSERT SET id=42
Sends the update →
Node2
INSERT SET id=42
← Sends the update
INSERT
6

Node1
INSERT SET id=42
Sends the update →
Receives the update →
Node2
INSERT SET id=42
← Sends the update
← Receives the update
INSERT
6

Node1
INSERT SET id=42
Sends the update →
Receives the update →
Duplicate key error!
Node2
INSERT SET id=42
← Sends the update
← Receives the update
Duplicate key error!
INSERT
6

Node1
DELETE WHERE id=42
Node2
DELETE WHERE id=42
DELETE
7

Node1
DELETE WHERE id=42
Sends the update →
Node2
DELETE WHERE id=42
← Sends the update
DELETE
7

Node1
DELETE WHERE id=42
Sends the update →
Receives the update →
Node2
DELETE WHERE id=42
← Sends the update
← Receives the update
DELETE
7

Node1
DELETE WHERE id=42
Sends the update →
Receives the update →
Key not found error!
Node2
DELETE WHERE id=42
← Sends the update
← Receives the update
Key not found error!
DELETE
7

Node1
UPDATE SET i=42
Node2
UPDATE SET i=25
UPDATE
8

Node1
UPDATE SET i=42
Sends the update →
Node2
UPDATE SET i=25
← Sends the update
UPDATE
8

Node1
UPDATE SET i=42
Sends the update →
Receives the update →
Node2
UPDATE SET i=25
← Sends the update
← Receives the update
UPDATE
8

Node1
UPDATE SET i=42
Sends the update →
Receives the update →
i=25
Node2
UPDATE SET i=25
← Sends the update
← Receives the update
i=42
UPDATE
8

• Each node for its own purpose
Application 1 Application 2
Data Distribution
10

• Multiple machines for better scalability
Node 1 Node 2
[Read] Scale
11

• Nodes in diﬀerent geographical regions
Belgium Brazil
GEO Distribution
12

• Waits when the main master deads
• Then accepts writes
Active Hot Standby
Hot Standby
13

Node1
UPDATE ...
Node2,3,...
Fully Synchronous Cluster
15

Node1
UPDATE ...
Sends the packet →
Node2,3,...
Fully Synchronous Cluster
15

Node1
UPDATE ...
Sends the packet →
Waits
Node2,3,...
← Receives the packet
Fully Synchronous Cluster
15

Node1
UPDATE ...
Sends the packet →
Waits
Waits
Node2,3,...
← Receives the packet
Applies changes
Fully Synchronous Cluster
15

Node1
UPDATE ...
Sends the packet →
Waits
Waits
Waits
Node2,3,...
← Receives the packet
Applies changes
← Conﬁrms
Fully Synchronous Cluster
15

Node1
UPDATE ...
Sends the packet →
Waits
Waits
Waits
Receives conﬁrmation →
Node2,3,...
← Receives the packet
Applies changes
← Conﬁrms
Fully Synchronous Cluster
15

Node1
UPDATE ...
Sends the packet →
Waits
Waits
Waits
Receives conﬁrmation →
Ready for new update
Node2,3,...
← Receives the packet
Applies changes
← Conﬁrms
Fully Synchronous Cluster
15

• Speed of the slowest member
Synchronization Penalty
16

• Speed of the slowest member
• Writes or reads should wait
Synchronization Penalty
16

• Speed of the slowest member
• Writes or reads should wait
• Tunable
Synchronization Penalty
16

• Speed of the slowest member
• Writes or reads should wait
• Tunable
• Galera
• wsrep sync wait
• InnoDB Cluster
• group replication consistency
Synchronization Penalty
16

• Synchronization takes too much time
• Nodes often disconnect
GEO Replication is Painful
17

How to Setup Safe Asynchronous Master-Master

• Business logic
• Databases
• Tables
• Rows
Divide and Rule
19

• Each server writes to its own set of tables
Node 1 Node 2
D1
D2
D3
D4
D1
D2
D3
D4
Databases and Tables
20

• Agreement on which rows
Node 1 Node 2
Row 1
Row 2
Row 3
Row 4
Row 1
Row 2
Row 3
Row 4
Rows
21

• Agreement on which rows
•
E.g. AUTO INCREMENT
• auto increment increment=NUMBER OF SERVERS
• auto increment offset = server id
• Custom index
• Create your own unique pattern for the each node
Rows
21

• Hot Standby
•
Easy to setup and maintain
•
Separating writes by database/table
• Comparatively easy to setup and maintain
• Separating writes by row
• Should be used very carefully
Conclusion
22

MySQL User Reference Manual
Why MySQL Replication Fails, and How to...
MySQL High Availability
More information
23

www.slideshare.net/SvetaSmirnova
twitter.com/svetsmirnova
github.com/svetasmirnova
Thank you!
24

When it comes to optimizing the performance of a database, DBAs have to look at everything from the OS to the network. In this session, MariaDB Enterprise Architect Manjot Singh shares best practices for getting the most out of MariaDB. He highlights recommended OS settings, important configuration and tuning parameters, options for improving replication and clustering performance and features such as query result caching.

How to set up orchestrator to manage thousands of MySQL serversSimon J Mudd

This document discusses how to scale Orchestrator to manage thousands of MySQL servers. It describes how Booking.com uses Orchestrator to manage their thousands of MySQL servers. As the number of monitored servers increases, integration with internal infrastructure is needed, Orchestrator performance must be optimized, and high availability and wider user access features are added. The document provides examples of configuration settings and special considerations needed to effectively use Orchestrator at large scale.

Maxscale_메뉴얼NeoClova

MaxScale is a database proxy that provides high availability and scalability for MariaDB servers. It can be used to configure load balancing of read/write connections, auto failover/switchover/rejoin using MariaDB GTID replication. Keepalived can be used with MaxScale to provide high availability by monitoring MaxScale and failing over if needed. The document provides details on setting up MariaDB replication with GTID, installing and configuring MaxScale and Keepalived. It also describes testing the auto failover functionality.

How to Manage Scale-Out Environments with MariaDB MaxScaleMariaDB plc

MaxScale is a database proxy that provides load balancing, connection pooling, and replication capabilities for MariaDB and MySQL databases. It can be used to scale databases horizontally across multiple servers for increased performance and availability. The document provides an overview of MaxScale concepts and capabilities such as routing, filtering, security features, and how it can be used for operational tasks like query caching, logging, and data streaming. It also includes instructions on setting up MaxScale with a basic example of configuring read/write splitting between a master and slave database servers.

MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11Kenny Gryp

Oracle's MySQL solutions make it easy to setup various database architectures and achieve high availability with the introduction MySQL InnoDB Cluster and MySQL InnoDB ReplicaSet meeting various high availability requirements. MySQL InnoDB ClusterSet provides a popular disaster recovery solution. Completely built in-house and supported by Oracle, many enterprises large and small have adopted these solutions into business critical applications. In this presentation the various database architecture solutions for high availability and disaster recovery will be covered and help you choose the right solutions based on your business requirements.

ProxySQL High Avalability and Configuration Management OverviewRené Cannaò

The document provides an overview of high availability and configuration management options for ProxySQL. It discusses deploying ProxySQL locally on application servers, in a dedicated layer, or using both approaches. When deploying in a dedicated layer, options for high availability include keepalived, load balancers, Consul, and Kubernetes. Configuration can be managed through tools like Ansible, Puppet, or by loading SQL files. ProxySQL Cluster enables syncing configuration across nodes.

MariaDB Performance Tuning and OptimizationMariaDB plc

This document discusses MariaDB performance tuning and optimization. It covers common principles like tuning from the start of application development. Specific topics discussed include server hardware, OS settings, MariaDB configuration settings like innodb_buffer_pool_size, database design best practices, and query monitoring and tuning tools. The overall goal is to efficiently use hardware resources, ensure best performance for users, and avoid outages.

[2018] MySQL 이중화 진화기NHN FORWARD

24시간 365일 서비스를 위한 MySQL DB 이중화. MySQL 이중화 방안들에 대해 알아보고 운영하면서 겪은 고민들을 이야기해 봅니다. 목차 1. DB 이중화 필요성 2. 이중화 방안 - HW 이중화 - MySQL Replication 이중화 3. 이중화 운영 장애 4. DNS와 VIP 5. MySQL 이중화 솔루션 비교 대상 - MySQL을 서비스하고 있는 인프라 담당자 - MySQL 이중화에 관심 있는 개발자

MariaDB 10.11 key features overview for DBAsFederico Razzoli

Webinar: MariaDB 10.11 key features overview for DBAs Orgnised by Vettabase 27 April 2023 Amongst other topics: - Long ALTER TABLES now don’t cause replicas to lag - InnoDB configuration is now more dynamic, and certain important variables can be modified without a restart - Populating an empty table is now much faster - New data types: UUID, INET4, INET6 - SFORMAT() function, NATURAL_KEY_SORT() function

PostgreSQL at 20TB and BeyondChris Travers

Highly efficient backups with percona xtrabackupNilnandan Joshi

Percona XtraBackup is an open source, free MySQL hot backup software that performs non-blocking backups for InnoDB and XtraDB databases. In this talk we'll describe below things. - How it works with MySQL/Percona Server and what are the features provided - Difference between Xtrabackup and Innobackupex - How to take full/increment/partial backup and restore - How to use features like streaming, compression, remote and compact backups - How to troubleshoot the issue with xtrabackup

Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp

This document provides a hands-on tutorial for advanced Percona XtraDB Cluster users. It discusses setting up a 3 node PXC cluster environment in VirtualBox and bootstrapping the initial cluster. It then covers topics like avoiding state snapshot transfers when restarting MySQL, recovering from clean and unclean shutdowns, and reproducing and diagnosing different types of conflicts through examples.

Maxscale 소개 1.1.1NeoClova

Disaster Recovery Planning for MySQL & MariaDBSeveralnines

Bart Oles - Severalnines AB Organizations need an appropriate disaster recovery plan to mitigate the impact of downtime. But how much should a business invest? Designing a highly available system comes at a cost, and not all businesses and indeed not all applications need five 9's availability. We will explain fundamental disaster recovery concepts and walk you through the relevant options from the MySQL & MariaDB ecosystem to meet different tiers of disaster recovery requirements, and demonstrate how to automate an appropriate disaster recovery plan.

1.5 literal sql & bind variable sql탑크리에듀(구로디지털단지역3번출구 2분거리)

리터럴(Literal) SQL이란? SQL문장 작성시 WHERE절의 비교되는 값에 문자/숫자 상수값을 하드코딩해서 작성한 것을 말하며, Bind Variable SQL이란 WHERE절의 특정값을 표시하는 자리에 바인드 변수 형태(:B)로 표시한 것을 말한다. 리터럴 SQL문을 많이 사용하면 하드파싱의 빈도를 높이게 되어 Library Cache 내에서 Cache되는 SQL문들이 자주 age out 하게 되므로 주기를 빠르게 하고 Dictionary Cache의 사용률을 높이게 된다. 이러한 Shared SQL Area의 SQL문 중에서 리터럴 SQL 문들을 찾아서 Bind Variable을 이용한 방법으로 바꾸어야 성능향상에 도움이 된다.

PostgreSQL for Oracle Developers and DBA'sGerger

This document provides information about PostgreSQL and Command Prompt Inc. It discusses that Command Prompt offers professional services for migrating from Oracle to PostgreSQL. It then covers aspects of PostgreSQL like its licensing, large international community, low costs, mature codebase, enterprise features, and technical capabilities like its SQL implementation, replication, foreign data wrappers and user-defined functions. It also notes tools that can help ease migration from Oracle and some differences from Oracle.

Building Stream Infrastructure across Multiple Data Centers with Apache KafkaGuozhang Wang

To manage the ever-increasing volume and velocity of data within your company, you have successfully made the transition from single machines and one-off solutions to large distributed stream infrastructures in your data center, powered by Apache Kafka. But what if one data center is not enough? I will describe building resilient data pipelines with Apache Kafka that span multiple data centers and points of presence, and provide an overview of best practices and common patterns while covering key areas such as architecture guidelines, data replication, and mirroring as well as disaster scenarios and failure handling.

MySQL 5.5 Guide to InnoDB StatusKarwin Software Solutions LLC

The Full MySQL and MariaDB Parallel Replication TutorialJean-François Gagné

MariaDB 10.0 introduces domain-based parallel replication which allows transactions in different domains to execute concurrently on replicas. This can result in out-of-order transaction commit. MariaDB 10.1 adds optimistic parallel replication which maintains commit order. The document discusses various parallel replication techniques in MySQL and MariaDB including schema-based replication in MySQL 5.6 and logical clock replication in MySQL 5.7. It provides performance benchmarks of these techniques from Booking.com's database environments.

Automated master failoverYoshinori Matsunobu

Automated, Non-Stop MySQL Operations and Failover discusses automating master failover in MySQL to minimize downtime. The goal is to have no single point of failure by automatically promoting a slave as the new master when the master goes down. This is challenging due to asynchronous replication and the possibility that not all slaves have received the same binary log events from the crashed master. Differential relay log events must be identified and applied to bring all slaves to an eventually consistent state.

Implementing Highly Performant Distributed AggregatesScyllaDB

Designing how to implement aggregates in a distributed database is a non-trivial task. When dealing with aggregates that will be polling the entire cluster, it is important to consider performance impacts. If done poorly, full table scans can bring production systems to their knees. So how can you implement aggregate functions without hammering real-time availability and performance for other read/write operations? Learn how distributed aggregates were implemented in ScyllaDB to balance performance across large NoSQL distributed database clusters.

MariaDB Galera ClusterAbdul Manaf

Galera cluster for high availability Mydbops

State transfer With GaleraMydbops

This document summarizes information about state transfers in Galera, an open source replication plugin for MySQL. It discusses when state transfers are needed, such as during network partitions or node failures. It describes the two types of state transfers - incremental state transfers (IST), which sync nodes incrementally, and state snapshot transfers (SST), which transfer a full data copy. The document compares different methods for SST, such as mysqldump, RSYNC, Xtrabackup, and Clone, and discusses their speeds, impact on donor nodes, and other characteristics. Overall, the document provides an overview of state transfers in Galera synchronous replication.

Galera cluster for MySQL - Introduction SlidesSeveralnines

Accelerating Ceph with RDMA and NVMe-oFinside-BigData.com

Haodong Tang from Intel gave this talk at the 2018 Open Fabrics Workshop. "Efficient network messenger is critical for today’s scale-out storage systems. Ceph is one of the most popular distributed storage system providing a scalable and reliable object, block and file storage services. As the explosive growth of Big Data continues, there're strong demands leveraging Ceph build high performance & ultra-low latency storage solution in the cloud and bigdata environment. The traditional TCP/IP cannot satisfy this requirement, but Remote Direct Memory Access (RDMA) can. "In this session, we'll present the challenges in today's distributed storage system posed by network messenger with the profiling results of Ceph All Flash Array system showing the networking already become the bottleneck and introduce how we achieved 8% performance benefit with Ethernet RDMA protocol iWARP. We'll first present the design of integrating iWARP to Ceph networking module together with performance characterization results with iWARP enabled IO intensive workload. The send part, we will explore the proof-of-concept solution of Ceph on NVMe over iWARP to build high-performance and high-density storage solution. Finally, we will showcase how these solutions can improve OSD scalability, and what’s the next optimization opportunities based on current analysis." Watch the video: https://wp.me/p3RLHQ-ikV Learn more: http://intel.com and https://insidehpc.com/2018/04/amazon-libfabric-case-study-flexible-hpc-infrastructure/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

MySQL Server Settings Tuningguest5ca94b

The document discusses tuning MySQL server settings for performance. Some key points covered include: - Settings are workload-specific and depend on factors like storage engine, OS, hardware. Tuning involves getting a few settings right rather than maximizing all settings. - Monitoring tools like SHOW STATUS, SHOW INNODB STATUS, and OS tools can help evaluate performance and identify tuning opportunities. - Memory allocation and settings like innodb_buffer_pool_size, key_buffer_size, query_cache_size are important to configure based on the workload and available memory.

Lessons Learned: Troubleshooting ReplicationSveta Smirnova

This document provides an overview of troubleshooting replication in MySQL and MariaDB databases. It discusses typical errors seen with the slave IO thread and slave SQL thread, such as replication stopping, the slave lagging behind the master, and the master increasing in resource usage. It also covers replication concepts like the master-slave architecture, binary logging formats, global transaction identifiers, and tools for monitoring replication like SHOW SLAVE STATUS and Performance Schema.

How Safe is Asynchronous Master-Master Setup?Sveta Smirnova

Presented at Percona MySQL Tech Day on September 10, 2020: https://www.percona.com/tech-days#mysql It is common knowledge that built-in asynchronous active-active replication is not safe. I remember times when the official MySQL User Reference Manual stated that such an installation is not recommended for production use. Some experts repeat this claim even now. While this statement is generally true, I worked with thousands of shops that successfully avoided asynchronous replication limitations in active-active setups. In this talk, I will show how they did it, demonstrate situations when asynchronous source-source replication is the best possible high availability option and beats such solutions as Galera or InnoDB Clusters. I will also cover common mistakes, leading to disasters.

Spil Games @ FOSDEM: Galera Replicator IRLspil-engineering

This document discusses Spil Games' use of Galera Replicator to improve the high availability and scalability of their MySQL databases. Some key points: - Spil Games is migrating legacy master-master MySQL databases to Galera for synchronous replication and high availability. - Their SSP (Storage Platform) is being moved from asynchronous master-master to Galera to avoid single points of failure. - Challenges included backup procedures, flow control during replication, and schema changes during upgrades. - Future plans include using Galera with OpenStack for database-as-a-service and exploring WAN replication.

More Related Content

What's hot (20)

MariaDB 10.11 key features overview for DBAsFederico Razzoli

PostgreSQL at 20TB and BeyondChris Travers

Highly efficient backups with percona xtrabackupNilnandan Joshi

Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp

Maxscale 소개 1.1.1NeoClova

Disaster Recovery Planning for MySQL & MariaDBSeveralnines

1.5 literal sql & bind variable sql탑크리에듀(구로디지털단지역3번출구 2분거리)

PostgreSQL for Oracle Developers and DBA'sGerger

Building Stream Infrastructure across Multiple Data Centers with Apache KafkaGuozhang Wang

MySQL 5.5 Guide to InnoDB StatusKarwin Software Solutions LLC

The Full MySQL and MariaDB Parallel Replication TutorialJean-François Gagné

Automated master failoverYoshinori Matsunobu

Implementing Highly Performant Distributed AggregatesScyllaDB

MariaDB Galera ClusterAbdul Manaf

Galera cluster for high availability Mydbops

State transfer With GaleraMydbops

Galera cluster for MySQL - Introduction SlidesSeveralnines

Accelerating Ceph with RDMA and NVMe-oFinside-BigData.com

MySQL Server Settings Tuningguest5ca94b

Lessons Learned: Troubleshooting ReplicationSveta Smirnova

MariaDB 10.11 key features overview for DBAsFederico Razzoli

PostgreSQL at 20TB and BeyondChris Travers

Highly efficient backups with percona xtrabackupNilnandan Joshi

Advanced Percona XtraDB Cluster in a nutshell... la suiteKenny Gryp

Maxscale 소개 1.1.1NeoClova

Disaster Recovery Planning for MySQL & MariaDBSeveralnines

1.5 literal sql & bind variable sql탑크리에듀(구로디지털단지역3번출구 2분거리)

PostgreSQL for Oracle Developers and DBA'sGerger

Building Stream Infrastructure across Multiple Data Centers with Apache KafkaGuozhang Wang

MySQL 5.5 Guide to InnoDB StatusKarwin Software Solutions LLC

The Full MySQL and MariaDB Parallel Replication TutorialJean-François Gagné

Automated master failoverYoshinori Matsunobu

Implementing Highly Performant Distributed AggregatesScyllaDB

MariaDB Galera ClusterAbdul Manaf

Galera cluster for high availability Mydbops

State transfer With GaleraMydbops

Galera cluster for MySQL - Introduction SlidesSeveralnines

Accelerating Ceph with RDMA and NVMe-oFinside-BigData.com

MySQL Server Settings Tuningguest5ca94b

Lessons Learned: Troubleshooting ReplicationSveta Smirnova

Similar to How Safe is Asynchronous Master-Master Setup? (20)

How Safe is Asynchronous Master-Master Setup?Sveta Smirnova

Spil Games @ FOSDEM: Galera Replicator IRLspil-engineering

Oss4b - pxc introductionFrederic Descamps

This document provides an introduction to parallel synchronous replication using Percona XtraDB Cluster (PXC). It discusses the limitations of traditional MySQL replication and how PXC implements a data-centric approach with synchronous multi-master replication between nodes. Key features of PXC highlighted include parallel replication, data consistency, and automatic provisioning of new nodes. The document also covers integration with load balancers and limitations to be aware of for write-intensive or large transaction workloads.

Inside the IT Territory game server / Mark Lokshin (IT Territory)DevGAMM Conference

In my talk, I will talk about the internal structure of the game server, which support our games of the Vorodezh ITT division. What can be learned from the talk: - what technologies did we choose for the development of game servers (spoiler: Vert.X Hazelcast, Postgres, Kafka, Prometheus + Grafana, Consul, Photon Cloud); - how we use them (spoiler: not all for their intended purpose); - how we install updates; - some interesting mistakes that we caught while working with Vert.X and Hazelcast.

1212312232maclean liu

Percon XtraDB Cluster in a nutshellFrederic Descamps

Percona XtraDB Cluster is a high availability and high scalability solution for MySQL clustering. Percona XtraDB Cluster integrates Percona Server with the Galera synchronous replication library in a single product package which enables you to create a cost-effective MySQL cluster. This tutorial will cover the following topics: - Migration from standard MySQL Master-Slave Architecture to PXC - Configuration differences between standard MySQLl and Xtradb Cluster - How to add a node and what does SST, IST mean ? How to use them ? - How to backup the cluster - How to monitor the cluster - 2 nodes servers- Why this isn't ideal but reasons and steps to setting it up anyway. - Galera Arbitrator: Defining what it is. - How to maintain the cluster - Setting up load balancing for Xtradb cluster - How to handle the cluster in the cloud - Tips and tricks - ... and if available cover PXC 5.6 with Galera 3 !!

stackconf 2023 | Rolling Updates: Database Version Migrations with minimum Di...NETWAYS

YDB is an open-source Distributed SQL database available under Apache 2.0 license. It’s easily scalable across thousands of nodes and is known to be always available. Version migration is a natural process for every software system. In majority of use cases YDB is used as a mission critical OLTP database that cannot afford maintenance windows and must remain available during version migrations. In this talk we will briefly describe YDB layered architecture and share some tricks to minimize database unavailability during minor and major versions migrations from YDB server and applications point of view.

Scylla on Kubernetes: Introducing the Scylla OperatorScyllaDB

The document introduces the Scylla Operator for Kubernetes, which provides a management layer for Scylla on Kubernetes. It addresses some limitations of using StatefulSets alone to run Scylla, such as safe scale down operations and tracking member identity. The operator implements the controller pattern with custom resources to deploy and manage Scylla clusters on Kubernetes. It handles tasks like cluster creation and scale up/down while addressing issues like local storage failures.

Forge blockchain deployment made easyArcBlock

Percona XtraDB ClusterKenny Gryp

Built-in MySQL Replication is known for its capability to enable to scale reads easily. However, there are some limitations and known issues with this solution because of the asynchronous nature of this replication. This talk will describe another way of doing MySQL replication, by using synchronous replication, available in Percona XtraDB Cluster. The open source solution will be explained and compared to traditional asynchronous MySQL replication, as well as some known use cases will be described. Percona XtraDB Cluster is an, open source, high availability and high scalability solution for MySQL clustering. Features include: Synchronous replication, Multi-master replication support, Parallel replication, Automatic node provisioning.

Cassandra for the ops dos and dontsDuyhai Doan

Scylla Summit 2018: Make Scylla Fast Again! Find out how using Tools, Talent,...ScyllaDB

Scylla strives to deliver high throughput at low, consistent latencies under any scenario. But in the field things can and do get slower than one would like. Some of those issues come from bad data modelling and anti-patterns. Some others from lack of resources and bad system configuration, and in rare cases even product malfunction. But how to tell them apart? And once you do, how to understand how to fix your application or reconfigure your system? Scylla has a rich ecosystem of tools available to answer those questions and in this talk we’ll discuss the proper use of some of them and how to take advantage of each tool’s strength. We will discuss real examples using tools like CQL tracing, nodetool commands, the Scylla monitor and others.

ScyllaDB Leaps Forward with Dor Laor, CEO of ScyllaDBScyllaDB

Kafka eosNitin Kumar

1) Apache Kafka is a distributed streaming platform that allows for continuous stream processing. It provides APIs for producers and consumers to read and write streams, as well as connectors and stream processing capabilities. 2) Kafka faces two main problems: duplicate writes and ensuring exactly-once processing. It solves duplicate writes through an idempotent producer that assigns unique IDs to each message. 3) Kafka solves exactly-once processing through transactions. Producers can initiate transactions to send multiple records atomically, and consumers only see committed transactions, ensuring exactly-once semantics for stream processing applications.

Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScyllaDB

mtl_rubykaigiHirotomo Oi

This document summarizes the experiences of operating Rails websites at stable scale. It discusses using virtualization to run multiple websites on a single server. It provides tips for optimizing performance including database tuning, adding indexes, log rotation, and restarting Mongrel processes periodically. New Rails sites are launched using the newest framework versions and different web servers like Passenger are tested for performance.

BlueHat v18 || Hardening hyper-v through offensive security researchBlueHat Security Conference

Jugal Parikh, Microsoft Holly Stewart, Microsoft Humans are susceptible to social engineering. Machines are susceptible to tampering. Machine learning is vulnerable to adversarial attacks. Singular machine learning models can be “gamed” leading to unexpected outcomes. In this talk, we’ll compare the difficulty of tampering with cloud-based models and client-based models. We then discuss how we developed stacked ensemble models to make our machine learning defenses less susceptible to tampering and significantly improve overall protection for our customers. We talk about the diversity of our base ML models and technical details on how they are optimized to handle different threat scenarios. Lastly, we’ll describe suspected tampering activity we’ve witnessed using protection telemetry from over half a billion computers, and whether our mitigation worked.

Why MySQL Replication Fails, and How to Get it BackSveta Smirnova

OpenStack Scale-out Networking ArchitectureRandy Bias

(1) The document describes OpenStack networking architecture that scales to over 1,000 servers without using Neutron. It uses a layer 3 spine and leaf topology with no VLANs. (2) Key components include Nova network which is distributed, NAT services on the edge, and a VIF driver that sets up routing and DHCP for each VM. (3) Challenges included moving OpenStack targets and an immature Neutron. The architecture provides a solid underlay for SDN and future integration with Neutron.

Preventing and Resolving MySQL DowntimeJervin Real

This document summarizes a presentation about preventing and resolving MySQL downtime. It discusses common issues that can cause downtime like disk failures, crashes from bugs or upgrades, unnoticed cluster node failures, and performance problems. Solutions presented include tools from Percona like XtraDB Cluster for high availability, the Percona Monitoring Plugins, and the Percona Toolkit. The presentation also covers preventing issues through monitoring, proper configuration of MySQL, the operating system, and hardware.

How Safe is Asynchronous Master-Master Setup?Sveta Smirnova

Spil Games @ FOSDEM: Galera Replicator IRLspil-engineering

Oss4b - pxc introductionFrederic Descamps

Inside the IT Territory game server / Mark Lokshin (IT Territory)DevGAMM Conference

1212312232maclean liu

Percon XtraDB Cluster in a nutshellFrederic Descamps

stackconf 2023 | Rolling Updates: Database Version Migrations with minimum Di...NETWAYS

Scylla on Kubernetes: Introducing the Scylla OperatorScyllaDB

Forge blockchain deployment made easyArcBlock

Percona XtraDB ClusterKenny Gryp

Cassandra for the ops dos and dontsDuyhai Doan

Scylla Summit 2018: Make Scylla Fast Again! Find out how using Tools, Talent,...ScyllaDB

ScyllaDB Leaps Forward with Dor Laor, CEO of ScyllaDBScyllaDB

Kafka eosNitin Kumar

Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScyllaDB

mtl_rubykaigiHirotomo Oi

BlueHat v18 || Hardening hyper-v through offensive security researchBlueHat Security Conference

Why MySQL Replication Fails, and How to Get it BackSveta Smirnova

OpenStack Scale-out Networking ArchitectureRandy Bias

Preventing and Resolving MySQL DowntimeJervin Real

More from Sveta Smirnova (20)

MySQL 2024: Зачем переходить на MySQL 8, если в 5.х всё устраивает?Sveta Smirnova

25 октябрая 2023 года Oracle прекратила активную поддержку MySQL 5.7. Это значит, что стоит присмотреться к улучшениям в версии 8: - Новому системному словарю - Современному SQL - Поддержке JSON, NoSQL, MySQL Shell, и возможности работать с MySQL как с MongoDB - Улучшениям в оптимизаторе запросов и диагностике Мой доклад для разработчиков приложений под MySQL. Я не буду рассказывать как конфигурировать сервер и сфокусируюсь на его использовании.

Database in Kubernetes: Diagnostics and MonitoringSveta Smirnova

Kubernetes is the new cool in 2023. Many database installations are on Kubernetes now. And this creates challenges for Support engineers because traditional monitoring and diagnostic tools work differently on bare hardware and Kubernetes. In this session, I will focus on differences in methods we use to collect metrics, describe challenges that Percona Support hits when working with database installations on Kubernetes, and discuss how we resolve them. This talk will cover all database technologies we support: MySQL, MongoDB, and PostgreSQL. Presented at Percona Live 2023

MySQL Database Monitoring: Must, Good and Nice to HaveSveta Smirnova

It is very easy to find if a database installation is having issues. You only need to enable Operating System monitoring. A disk, memory, or CPU usage change will alert you about the problems. But they would not show *why* the trouble happens. You need the help of database-specific monitoring tools. As a Support Engineer, I am always very upset when handling complaints about the database behavior lacking specific database monitoring data because I cannot help! There are two reasons database and system administrators do not enable necessary instrumentation. The first is a natural or expected performance impact. Second is the lack of knowledge on what needs to be on to resolve a particular issue. In this talk, I will cover both concerns. I will show which monitoring instruments will give information on what causes disk, memory, or CPU problems. I will teach you how to use them. I will uncover which performance impact these instruments have. I will use both MySQL command-line client and open-source graphical instrument Percona Monitoring and Management (PMM) for the examples.

MySQL Cookbook: Recipes for DevelopersSveta Smirnova

This document provides an overview of the MySQL Cookbook by O'Reilly. It discusses the intended audience of database administrators and developers. It also demonstrates different ways of interacting with MySQL, including through the command line interface, MySQL Shell, and X DevAPI. Examples are provided for common tasks like reading, writing, and updating data in both standard SQL and the object-oriented X DevAPI.

MySQL Performance for DevOpsSveta Smirnova

MySQL performance can be improved by tuning queries, server options, and hardware. Traditionally it was an area of responsibility for three different roles: Development, DBA, and System Administrators. Now DevOps handle these all. But there is a gap. Knowledge gained by MySQL DBAs after years or focusing on a single product is hard to gain when you focus on more than one. This is why I am doing this session. I will show a minimal but most effective set of options to improve MySQL performance. For illustrations, I will use real user stories gained from my Support experience and Percona Kubernetes operators for PXC and MySQL.

MySQL Test Framework для поддержки клиентов и верификации баговSveta Smirnova

Talk for TestDriven Conf: https://tdconf.ru/2022/abstracts/8763 MySQL Test Framework (MTR) — это фреймворк для регрессионных тестов MySQL. Тесты для него пишут разработчики MySQL и запускаются во время подготовки к новым релизам. MTR можно использовать и по-другому. Я его использую, чтобы тестировать проблемы, о которых сообщают клиенты, и подтверждать сообщения об ошибках (bug reports) одновременно на нескольких версиях MySQL. При помощи MTR можно: * программировать сложные развёртывания; * тестировать проблему на нескольких версиях MySQL/Percona/MariaDB-серверов при помощи одной команды; * тестировать несколько одновременных соединений; * проверять ошибки и возвращаемые значения; * работать с результатами запросов, хранимыми процедурами и внешними командами. Тест может быть запущен на любой машине с MySQL-, Percona- или MariaDB-сервером. Я покажу, как я работаю с MySQL Test Framework, и надеюсь, что вы тоже полюбите этот инструмент.

MySQL Cookbook: Recipes for Your BusinessSveta Smirnova

This document provides an overview of different ways to work with MySQL using standard SQL, X DevAPI, and MySQL Shell utilities. It discusses querying, updating, and exporting/importing data using these different approaches. It also covers topics like character encoding, generating summaries, storing errors, and retrieving metadata. Examples are provided to illustrate concepts like selecting, grouping, joining, changing data, common table expressions, and more using SQL and X DevAPI. MySQL Shell utilities for exporting/importing CSV, JSON, and working with collections are also demonstrated.

Introduction into MySQL Query Tuning for Dev[Op]sSveta Smirnova

Percona Live Online 2021 talk: https://www.percona.com/resources/videos/introduction-mysql-query-tuning-for-devops In this talk I will show how to get started with MySQL Query Tuning. I will make a short introduction into physical table structure and demonstrate how it may influence query execution time. Then we will discuss basic query tuning instruments and techniques, mainly EXPLAIN command with its latest variations. You will learn how to understand its output and how to rewrite queries or change table structure to achieve better performance.

Производительность MySQL для DevOpsSveta Smirnova

Talk for the DevOps Pro Moscow 2021: https://www.devopspro.ru/Sveta-Smirnova/ Производительность MySQL можно улучшить при помощи оптимизации запросов, настроек MySQL сервера и железа. Традиционно эти задачи распределялись между тремя ролями: Разработчик, Администратор баз данных и Системный Администратор. Теперь же все эти задачи решает DevOps, что непросто для одного человека. В этом докладе я расскажу об основных оптимизациях, которые решают большинство проблем производительности MySQL. Для иллюстраций я буду использовать реальные пользовательские истории и Percona Kubernetes Operator.

MySQL Performance for DevOpsSveta Smirnova

How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB ClusterSveta Smirnova

Percona XtraDB Cluster (PXC) is a 100% synchronized cluster in regards to DML operations. It is ensured by the optimistic locking model and ability to rollback transaction which cannot be applied on all nodes. However, DDL operations are not transactional in MySQL. This adds complexity when you need to change the schema of the database. Changes made by DDL may affect the results of the queries. Therefore all modifications must replicate on all nodes prior to the next data access. For operations that run momentarily, it can be easily achieved, but schema changes may take hours to apply. Therefore in addition to the safest synchronous blocking schema upgrade method: TOI, - PXC supports more relaxed, though not safe, method RSU. RSU: Rolling Schema Upgrade is advertised to be non-blocking. But you still need to take care of updates, running while you are performing such an upgrade. Surprisingly, even updates on not related tables and schema can cause RSU operation to fail. In this talk, I will uncover nuances of PXC schema upgrades and point to details you need to take special care about. Further Information Schema change is a frequent task, and many do not expect any surprises with it. However, the necessity to replay the changes to all synchronized nodes adds complexity. I made a webinar on a similar topic which was recorded and available for replay. Now I have found that I share a link to the webinar to my Support customers approximately once per week. Not having a good understanding of how schema change works in the cluster leads to lockups and operation failures. This talk will provide a checklist that will help to choose the best schema change method. Presented at Percona Live Online: https://perconaliveonline2020.sched.com/event/ePm2/how-to-avoid-pitfalls-in-schema-upgrade-with-percona-xtradb-cluster

How to migrate from MySQL to MariaDB without tearsSveta Smirnova

Presented at MariaDB Server Fest 2020: https://mariadb.org/fest2020/migrate-mysql/ MariaDB is a drop-in replacement for MySQL. Initial migration is simple: start MariaDB over the old MySQL datadir. Later your application may notice that some features work differently than with MySQL. These are MariaDB improvements, so this is good and, likely the reason you migrated. In this session, I will focus on the differences affecting application performance and behavior. In particular, features sharing the same name, but working differently.

Modern solutions for modern database load: improvements in the latest MariaDB...Sveta Smirnova

Presented at MariaDB Server Fest 2020: https://mariadb.org/fest2020/improvements/ MariaDB is famous for working well in high-performance environments. But our view of what to call high-performance changes over time. Every year we get faster data transfer speed; more devices connected to the Internet; more users and, as a result, more data. Challenges, which developers have to solve, are getting harder. This session shows what engineers do to keep the product up to date, focusing on MariaDB improvements that make it different from its predecessor, MySQL.

Современному хайлоду - современные решения: MySQL 8.0 и улучшения PerconaSveta Smirnova

MySQL всегда использовали под высокой нагрузкой. Недаром эта база была и остаётся самым популярным бэкэндом для web. Однако наши представления о хайлоде с каждым годом расширяются. Большая скорость передачи данных -> больше устройств с подключением к интернет -> больше пользователей -> больше данных. Задачи, стоящие перед разработчиками MySQL, с каждым годом усложняются. В этом докладе я расскажу как менялись сценарии использования MySQL за [почти] 25 лет её истории и что делали инженеры, чтобы MySQL оставалась актуальной. Мы затронем такие темы, как работа с большим количеством активных соединений и высокими объёмами данных. Я покажу насколько современные версии лучше справляются с возросшими нагрузками. Я надеюсь, что после моего доклада те слушатели, которые используют старые версии, захотят обновиться и те, кто уже обновились, узнают как использовать современный MySQL на полную мощность. Прочитана на конференции OST 2020: https://ostconf.com/materials/2857#2857

How to Avoid Pitfalls in Schema Upgrade with GaleraSveta Smirnova

This document discusses different methods for performing schema upgrades in a Galera cluster, including Total Order Isolation (TOI), Rolling Schema Upgrade (RSU), and the pt-online-schema-change tool. TOI blocks the entire cluster during DDL but ensures consistency, while RSU allows upgrades without blocking the cluster but requires stopping writes and carries risks of inconsistency. Pt-online-schema-change performs non-blocking upgrades using triggers to copy rows to a new table in chunks.

Introduction to MySQL Query Tuning for Dev[Op]sSveta Smirnova

To get data, we query the database. MySQL does its best to return requested bytes as fast as possible. However, it needs human help to identify what is important and should be accessed in the first place. Queries, written smartly, can significantly outperform automatically generated ones. Indexes and Optimizer statistics, not limited to the Histograms only, help to increase the speed of the query a lot. In this session, I will demonstrate by examples of how MySQL query performance can be improved. I will focus on techniques, accessible by Developers and DevOps rather on those which are usually used by Database Administrators. In the end, I will present troubleshooting tools which will help you to identify why your queries do not perform. Then you could use the knowledge from the beginning of the session to improve them.

Billion Goods in Few Categories: How Histograms Save a Life?Sveta Smirnova

We store data with an intention to use it: search, retrieve, group, sort... To do it effectively the MySQL Optimizer uses index statistics when compiles the query execution plan. This approach works excellently unless your data distribution is not even. Last year I worked on several tickets where data follow the same pattern: millions of popular products fit into a couple of categories and rest used the rest. We had a hard time to find a solution for retrieving goods fast. We offered workarounds for version 5.7. However new MariaDB and MySQL 8.0 feature: histograms, - would work better, cleaner and faster. The idea of the talk was born. Of course, histograms are not a panacea and do not help in all situations. I will discuss: how index statistics physically stored by the storage engine which data exchanged with the Optimizer why it is not enough to make correct index choice when histograms can help and when they cannot differences between MySQL and MariaDB histograms

A Billion Goods in a Few Categories: When Optimizer Histograms Help and When ...Sveta Smirnova

Last year this session’s speaker worked on several cases where data followed the same pattern: millions of popular products fit into a couple of categories, and the rest uses the rest. Her team had a hard time finding a solution for retrieving goods quickly. MySQL 8.0 has a feature that resolves such issues: optimizer histograms, storing statistics of an exact number of values in each data bucket. In real life, histograms don’t help with all queries accessing nonuniform data. How you write a statement, the number of rows in the table, data distribution: All of these may affect the use of histograms. This presentation shows examples demonstrating how the optimizer works in each case, describes how to create histograms, and covers differences between MySQL and Oracle implementations.

Что нужно знать о трёх топовых фичах MySQLSveta Smirnova

MySQL прочно удерживает второе по популярности место после Oracle в рейтинге DB-engines: https://db-engines.com/en/ranking_trend Репликация, табличные движки и поддержка NoSQL не дают MySQL сдавать позиции с 2012 года: года основания рейтинга. Что особенного в этих фичах? Что нужно знать, чтобы использовать их на полную мощность? Я расскажу про дизайн. Именно он отвечает за то, чтобы ваше приложение не достигло потолка производительности. Понимание архитектуры поможет при проектирование нового приложения, которое впоследствии будет легко масштабироваться. Доклад рассчитан для начинающих пользователей MySQL. Однако поможет освежить свои знания и более опытным.

Billion Goods in Few Categories: How Histograms Save a Life?Sveta Smirnova

We store data with an intention to use it: search, retrieve, group, sort... To do it effectively, the MySQL Optimizer uses index statistics when it compiles the query execution plan. This approach works excellently unless your data distribution is not even. Last year I worked on several support tickets where data follows the same pattern: millions of popular products fit into a couple of categories and the rest used the rest. We had a hard time finding a solution for retrieving goods fast. We offered workarounds for version 5.7. However, a new MariaDB and MySQL 8.0 feature - histograms - would work better, cleaner and faster. The idea of the talk was born. Of course, histograms are not a panacea and do not help in all situations. I will discuss - how index statistics physically stored by the storage engine - which data exchanged with the Optimizer - why it is not enough to make correct index choice - when histograms can help and when they cannot - differences between MySQL and MariaDB histograms Talk for Percona Live 2019 Austin: https://www.percona.com/live/19/sessions/billion-goods-in-few-categories-how-histograms-save-a-life