MySQL highav Availability

MySQL High Availability
Solutions Best Practices
baruch osoveskiy
baruch@brillix.co.il

About Me
Baruch osoveskiy
•Senior Consultant @ brillix.co.il
•Linux-Unix SYSADMIN form 1997
•DBA on Oracle and MySQL Since 2000
•Working with unstructured Data (“big data”) since
2000 like text and spatial images etc..
https://github.com/barucho
http://www.slideshare.net/baruchosoveskiy

• In the end will be Q&A slide –
you can ask question I will try to answer
• The Demo will be in github :
https://github.com/barucho/MySQLHA2016.git
• You can send me question :
baruch@brillix.com
You can find more presentations on
http://www.slideshare.net/baruchosoveskiy
• I write in ildba.co.il (not so often)

Redundancy
▪ multiple copies of data in event of a disaster or
bad DML/DDL run in Production
Scaling
▪ to increase throughput (more network, disk, cpu)
High Availability
▪ get databases back online quickly
High Availability , Redundancy, Scaling ?

CAP …
Consistency
▪ All nodes see the same data at the same time
• Availability
▪ Every request receives a response
about whether it succeeded or not (COMMIT ?)
• Partition Tolerance
▪ The system continues to operate
despite arbitrary partitioning due to
network failures

What is my expected UP-TIME (9`s)?

Manual Failover
▪ human usually can make a better decision as to
whether failover is necessary.
▪ Avoid ping-ping between nodes!
▪ easy to configure
Automatic Failover
▪ More 9’s due to minimized outages.
▪ No need to wait on a DBA/ops.
▪ Very Complicate to configure (need cluser/scripts)
Automatic or Manual Failover?

Galera Cluster
High Availability, Scaling-improve read performance
• Active Active MySQL* cluster
• On Commit Changes are collected into a write
set.
• Write set is set to all nodes for certification.
• PKs are used to determine if the write set can be
applied.
• If certification fails, the write set is dropped and
the transaction is rolled back.
• All nodes will reach the same transaction
eventually.

Corosync-pacemaker DRDB active/passive
High Availability
• Corosync/pacemaker cluster or a co-ordinating
the startup and recovery of inter-related services
across a set of machines.
• DRDB shared storage systems by networked
mirroring
• Only one mysqld process can run at a time
• No active slave.
• DRDB can have performance issues in high IO
load
• Complicate to Configure

Master Slave
DRDB
corosync
pacemaker
DRDB
MySQL MySQL
corosync pacemaker DRDB active/passive
MySQL
quorum

Master Slave
DRDB
corosync
pacemaker
DRDB
MySQL MySQL
MySQL
quorum
Master

Master Slave
DRDB
corosync
pacemaker
DRDB
MySQLMySQL
MySQL
quorum
MySQL

• One or more mysqld connected to one or more
data nodes
• The data on the data node is shard and
replicated
• Simple and automatic sharding and replication
• To use MySQL cluster you have to use NDB
storage engine.
MySQL Cluster / NDB / sharding
High Availability, Scaling

mysqld
MySQL Cluster / NDB / sharding
mysqld mysqld mysqld
A1
B2
B1
A2
NDB NDB NDB
NDB API
SQL
REST API

February 2015: MySQL Cluster 7.4 - 200 Million NoSQL QPS
MySQL Cluster 7.4 delivers massively concurrent NoSQL access –
200 Million reads per second using the FlexAsync benchmark.
This was achieved with 32 (out of a maximum 48) data nodes,
each running on a server with 2x Intel Haswell E5-2697 v3 CPUs.
https://www.mysql.com/why-mysql/benchmarks/mysql-cluster/

INNODB vs NDB
Feature InnoDB 1.1 NDB 7.5
Transactions All standard types READ COMMITTED
MVCC Yes No
Data Compression Yes No
Large Row Support (> 14K) Supported for VARBINARY,
VARCHAR, BLOB, and TEXT
columns
Supported for BLOB and TEXT
columns only
High Availability (HA) Requires additional software Yes (Designed for 99.999%
uptime)
Time for Node Failure Recovery 30 seconds or longer Typically < 1 second
NoSQL Access to Storage Engine Yes Yes

What is MySQL Replication
Master SlaveNETWORK

What is MySQL Replication
Master SlaveNETWORK
Slave
Slave
Slave
Slave

Why MySQL Replication
• High availability – can promote slave to
master
• Scale Out – add multiple read slaves
• Analytics – can run Analytics on live data.
• Backup – run cold backup on slave.
• For DEV and Pre-Prod

How MySQL Replication Works
Replication Formats
• SBR - Statement Based Replication
• Nondeterministic function problem
• RBR - Row Based Replication
• High network usage
• Higher IO usage
• Mixed Mode

Not safe Nondeterministic function
FOUND_ROWS(), GET_LOCK(), IS_FREE_LOCK(),
IS_USED_LOCK(), LOAD_FILE(), MASTER_POS_WAIT(),
RELEASE_LOCK(), ROW_COUNT(), SESSION_USER(),
SLEEP(), SYSDATE(), SYSTEM_USER(), USER(), UUID(),
and UUID_SHORT().

Where the Changes Stored
• Binary Log – Master (blue-bin.000002)
• Relay Log – Slave (white-relay-bin.000003)

Replication Threads
• Master Server
• Binlog Dump Thread
• Slave Server
• IO Thread
• SQL Thread

Method of Sending the Data (Replication Type)
• Asynchronous Replication
• Semi-Synchronous Replication (5.5 )
• Group Replication (plugin)

Master
TRX1
TRX2
TRX3
.
Slave
1
Binary
log
DML
+
COMMIT
Binlog Dump Thread

Master
TRX1
TRX2
TRX3
.
Slave
1
2
Binary
log
IO Thread
DML
+
COMMIT

Master
TRX1
TRX2
TRX3
.
Slave
TRX1
TRX2
TRX3
.
1
2
3
Binary
log
Relay
Log
IO Thread
DML
+
COMMIT

Master
TRX1
TRX2
TRX3
.
Slave
TRX1
TRX2
TRX3
.
1
2
3
4
Binary
log
Relay
Log
IO Thread
SQL Thread
DML
+
COMMIT

Master
TRX1
TRX2
TRX3
.
Slave
TRX1
TRX2
TRX3
.
1
2
3
4
5
Binary
log
Relay
Log
IO Thread
SQL Thread
DML
+
COMMIT

MySQL 5.6 Replication
What's New
• Multi threaded Slave
• Relay log recovery
• Binary Log Group Commit
• Crash-safe Replication – Binlog Checksums
• Global Transaction ID
You can read more in:
http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

Multi-Threaded Slaves
Replication performance is improved by using multiple
execution threads to apply replication events
slave-parallel-workers=2

Binary Log Group Commit
• Commit a group of transactions to the binary log.
• Ensures durability with good performance.
• Designed to handle seamlessly temporary peeks on the
workload.
enable by default in 5.6

Relay log recovery
• slave can automatically recover from a failure and resume
replicating DML updates
• the slave can roll back replication to the last successfully
committed transaction
relay-log-recovery =1
And in
Bug #13893363

Position (pre 5.6)
• Transactions are not guaranteed to execute in the order of
the Binary logs
• The show slave status is not accurate

Position
Master
Slave1
Slave
2
Binlog
File:mysql-bin.0010
Position:1238574
Binlog
File:mysql-bin.0020
Position:1043
Binlog
File:mysql-bin.0140
Position:8373

Position - What's the problem
• The same event can have different Binary log file and
position on different slaves
• Hard to find inconsistency in replication
• Very problematic to reconfigure replication

Global Transaction ID
• Unique identifier of a transaction across all servers in replication
topology
The GTID look like :
GTID = source_id:transaction_id
Where
transaction_id = transaction_start-transaction_stop
eg :
3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5

Master
Slave1
Slave
2
Binlog
File:mysql-bin.0010
3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5
Binlog
File:mysql-bin.0020
3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5
Binlog
File:mysql-bin.0140
3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5

How to enable GUID on MySQL replication
• Enable read only mode on Slaves
set global read_only=1
• Compare the binlog position on master and slave
show slave status
show master status
• Change configuration on all servers and restart
bin-log
guid_mode=ON
Enforce-guid-consistency
• Initialize GUID on all servers
change master to MASTER_AUTO_POSITION = 1;
• Disable read only mode on Slaves

Replication Setup
master.my.cnf
[mysqld]
...
server-id=1
report-host=blue
binlog-format=MIXED
log-bin
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
binlog-checksum=CRC32
master-verify-checksum=1

Replication Setup
slave.my.cnf
[mysqld]
...
server-id=2
report-host=white
binlog-format=MIXED
log-slave-updates=true
log-bin
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
slave-parallel-workers=2
binlog-checksum=CRC32
master-verify-checksum=1
slave-sql-verify-checksum=1
binlog-rows-query-log_events=1

Group Replication
• The replication group is a set of servers that interact with each other
through message passing
• Each server in the group may execute transactions independently
• RW transaction only commits after that operation is coordinated
within the group
• when a transaction is ready to commit, the server atomically
broadcasts the write values (row events) and write set (unique
identifiers of the rows that were updated). This establishes a global
total order for that transaction

Group Replication
• A highly available distributed MySQL database service
• Removes the need for manually handling server fail-over
• Provides distributed fault tolerance
• Enables Active/Active update anywhere setups
• Automates reconfiguration (adding/removing nodes, crashes,
failures)
• Automatically detects and handles conflicts

• Automated master monitoring and failover.
• MHA consists of MHA Manager and MHA Node
packages. MHA Manager runs on a manager server,
and MHA Node runs on each MySQL server
• MHA is a free opensource tool
• Can be configured to power off old Master to avoid
split brain.
MySQL Replication and MHA

MaxScale (open source database proxy)
• Open Source; Developed by mariadb
• “simple” database proxy
• Provides plugins
• Content aware

MaxScale (open source database proxy)
Redundancy, High Availability, Scaling (read)
Master Slave
MaxScle
MaxScle
MaxScle
Client
Read Write Read Only
Slave
Slave
MySQL
ReplicationSlave
Delay
Replication

MySQL Router
• Open Source; Developed by Oracle MySQL HA
• Simple layer 4 packet router
• No packet inspection
• Have Framework for plugins
• Natively integrated with MySQL Fabric
• Support MySQL InnoDB Cluster (Lab)

MySQL Router - plugins
• Routing
• First available
• Round robin
• Fabric cache
• Logger

fa fa
MySQL Router
Slave
MySQL
Router
Client
Slave
MySQL
Replication
Fabric
repository
RO HA GROUP
Metadata
RW HA GROUP

More database routers/Load balancer
• HAProxy
• F5 BigIP
• ProxySQL
• ELB (AWS)

MySQL INNODB Cluster
MySQL InnoDB Cluster: Vision “A single product — MySQL
— with high availability and scaling features baked in;
providing an integrated end-to-end solution that is easy to
use.”
Redundancy, High Availability, Scaling

MySQL INNODB Cluster
A minimum of three instances are required to create an InnoDB
cluster that is tolerant to the failure of one instance.

3306 3307
MaxScle
Client
3308
MySQL
Replication

Thank You

MySQL highav Availability

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to MySQL highav Availability (20)

Recently uploaded (20)

MySQL highav Availability