MySQL Group Replication

MySQL Group Replication

Kenny Gryp (@gryp)
1 / 65

Table of Contents
1. Overview 5. Backups
2. Provisioning Nodes 6. Load Balancers
3. Conﬁguration 7. Improvements
4. Monitoring
2 / 65

Apologies upfront. I was able to spend limited time researching
Group Replication, my knowledge is a lot more limited compared
to many MySQL (Group Replication) developers in this room. I
may have made wrong assumptions or discuss problems, missing
features which are likely known and on the roadmap to be
ﬁxed/developed in reasonable time. Group Replication is a quite
new feature and only recently became GA. Hereby I give full
responsibility to the MySQL Developers to respond whenever I
made a wrong statement :-), but please keep in mind that my talk is
only 25 minutes long. Even though MySQL Group Replication is
marked GA, it is still a new feature and database software adoption
usually takes a long time, bugs mentioned (some of which are not
veriﬁed yet) in these slides are not here to try to tell you the feature
is not good, I believe it is my duty as member of the community to
provide feedback and getting the opportunity to talk in this room
with a lot of Oracle employees clearly demonstrates this is their
desire as well, even though it does scare me quite a lot! This talk
describes the status of Group Replication on 31/01/2017.
3 / 65

Overview
4 / 65

MySQL Asynchronous Replication
5 / 65

6 / 65

Quick Overview
Writes in entire Group Replication executed in
'Global Total Order'
Majority consensus (Paxos Mencius)
Writes will be received/accepted by majority of the nodes
No guarantee all nodes have received a trx before
application gets OK back
Optimistic Locking: Conﬂict Detection after replicating trx:
'Certiﬁcation'
First Committer Wins
Every node has all data, cluster is 'as fast as slowest node'.
Nodes can join/leave cluster
7 / 65

Properties
No concept of master/slave, only 'members'
Durability:
No data loss, when failure of nodes happen. Does not accept
writes if there is no Quorum.
Active:Active Master:
All nodes can be conﬁgured to accept writes at same time *
No (time consuming) failover is necessary, every member can
become a writer member at any time
Added latency to every transaction COMMIT.
8 / 65

Main focus: Design & Usability
Performance & Stability was not yet analyzed
11 / 65

Use Cases for Group Replication
Environments with strict durability requirements
(no data loss if master member is lost)
Write to multiple nodes
('scalability' by splitting write/read workloads)
Improve failover time
...
12 / 65

Provisioning Nodes
13 / 65

GTID
Please note that Group Replication uses GTID
14 / 65

GTID
Please note that Group Replication uses GTID
Keep into Account:
Creating a cluster and provisioning nodes requires 'compatible'
GTID-sets
Errant Transactions!
15 / 65

Errant Transactions
Ensure there are no errant transactions before starting group
replication:
[ERROR] Plugin group_replication reported:
'This member has more executed transactions than those present
in the group.
Local transactions: 74dc6ab2-e1cc-11e6-92aa-08002789cd2e:1
>
Group transactions: 72149827-e1cc-11e6-9daf-08002789cd2e:1,
da7aba5e-dead-da7a-ba55-da7aba5e57ab:1-5'
[ERROR] Plugin group_replication reported:
'The member contains transactions not present in the group.
The member will now exit the group.'
[Note] Plugin group_replication reported:
'To force this member into the group you can use
the group_replication_allow_local_disjoint_gtids_join option'
16 / 65

Best Practice
Please never use
group_replication_allow_local_disjoint_gtids_join
once you use it, you always have to keep it on.
they might have been writes to the individual node (GR not
active)
data consistency/split brain/data loss/...
#84728: GR failure at start still starts MySQL
#84733: not possible to start with super_read_only=1
17 / 65

Starting Cluster
First Choose the right node to bootstrap:
mysql> select @@global.gtid_executedG
************************** 1. row ***************************
@@global.gtid_executed: 72149827-e1cc-11e6-9daf-08002789cd2e:1,
740e1fd2-e1cc-11e6-a8ec-08002789cd2e:1-2,
74dc6ab2-e1cc-11e6-92aa-08002789cd2e:1-2,
da7aba5e-dead-da7a-ba55-da7aba5e57ab:1-399
1 row in set (0.00 sec)
ensure compatible GTID sets or forget about it!
choose node with all GTIDs
18 / 65

Cluster Membership Operations
Start a new cluster:
SET GLOBAL group_replication_bootstrap_group=on;
START GROUP_REPLICATION;
SET GLOBAL group_replication_bootstrap_group=off;
19 / 65

Cluster Membership Operations
Start a new cluster:
SET GLOBAL group_replication_bootstrap_group=on;
SET GLOBAL group_replication_bootstrap_group=off;
Restore (GTID-enabled) Backup
SET GLOBAL group_replication_group_seeds='node1,node2,node3';
#84674: unresolved hostnames block GR from starting
20 / 65

Conﬁguration
21 / 65

Conﬁguration Requirements
[mysqld]
log-bin
binlog-format=row
binlog-checksum=NONE
gtid-mode=ON
log-slave-updates
master-info-repository=TABLE
relay-log-info-repository=TABLE
transaction-write-set-extraction=XXHASH64
Group Replication Conﬁguration:
group_replication_group_name="da7aba5e-dead-da7a-ba55-da7aba5e57ab"
group_replication_local_address= "gr-2:24901"
group_replication_group_seeds= "gr-1:24901,gr-2:24901,gr-3:24901"
22 / 65

MySQL InnoDB Cluster
No Security/Authentication is described in these slides
Possible to create a cluster in the MySQL Shell w. AdminAPI
Also performs conﬁguration checks
23 / 65

Other Requirements/Limitations
Required:
InnoDB Required
PK on every table
24 / 65

Other Requirements/Limitations
Required:
InnoDB Required
PK on every table
Not supported:
Transaction Savepoints
#84799: mysqldump --single-transaction uses
savepoints, does not work with GR
In multi-writer/active:active
Concurrent DDL vs DML/DDL operations
25 / 65

Oracle's (Valid) Recommendations
Only use
group_replication_single_primary_mode=ON
write to a single node only
Not recommended for WAN
(Probably because of Majority Consensus in Paxos Mencius)
Requires uneven amount of nodes for proper Quorum
26 / 65

My Recommendations - my.cnf
27 / 65

Do not use loose- even though it is mentioned in the manual
#84631: installation documentation issues
28 / 65

._allow_local_disjoint_gtids_join=OFF
29 / 65

Single writer mode?
group_replication_auto_increment_increment=7
default is too high. Set to 1.
30 / 65

Single writer mode?
group_replication_bootstrap_group=OFF
31 / 65

Single writer mode?
group_replication_bootstrap_group=OFF
group_replication_start_on_boot=ON
#84728: GR failure at start still starts MySQL 32 / 65

My Recommendations
Ensure all FQDN hostnames are resolvable.
@@global.hostname is used by other
members
34 / 65

My Recommendations
Ensure all FQDN hostnames are resolvable.
@@global.hostname is used by other
members
Dangerous to issue:
SET GLOBAL read_only=OFF;
SET GLOBAL super_read_only=OFF;
STOP GROUP_REPLICATION;
#84795: STOP GROUP_REPLICATION sets
super_read_only=off
35 / 65

My Failed Recommendation
In an attempt to prevent split brain and because of:
I tried to enforce super_read_only=1 at boot, but that
failed too:
I did not ﬁnd a way to prevent a MySQL node from starting as a
individual r/w MySQL server when Group Replication failed to
start.
36 / 65

Monitoring
(Proﬁling, Trending, Alerting, Status, Troubleshooting)
37 / 65

Performance Schema
SELECT TABLE_NAME FROM information_schema.TABLES
WHERE TABLE_SCHEMA='performance_schema'
AND TABLE_NAME LIKE '%replication%';
+-------------------------------------------+
| TABLE_NAME |
+-------------------------------------------+
| replication_applier_configuration |
| replication_applier_status |
| replication_applier_status_by_coordinator |
| replication_applier_status_by_worker |
| replication_connection_configuration |
| replication_connection_status |
| replication_group_member_stats |
| replication_group_members |
+-------------------------------------------+
2 replication appliers:
group_replication_applier <- group replication
38 / 65

Trending - SHOW GLOBAL
STATUS
Limited Status Information (which are usually easy to gather):
mysql> show global status like '%group%';
+----------------------------------+--------------------------------------+
| Variable_name | Value |
+----------------------------------+--------------------------------------+
| Com_group_replication_start | 0 |
| Com_group_replication_stop | 0 |
| group_replication_primary_member | 72149827-e1cc-11e6-9daf-08002789cd2e |
+----------------------------------+--------------------------------------+
39 / 65

Trending - PFS
mysql> select * from replication_group_member_statsG
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 14860449946972589:2
MEMBER_ID: 74dc6ab2-e1cc-11e6-92aa-08002789cd2e
COUNT_TRANSACTIONS_IN_QUEUE: 0 # Certification queue
COUNT_TRANSACTIONS_CHECKED: 4
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS:
72149827-e1cc-11e6-9daf-08002789cd2e:1,
740e1fd2-e1cc-11e6-a8ec-08002789cd2e:1-2,
74dc6ab2-e1cc-11e6-92aa-08002789cd2e:1-2,
da7aba5e-dead-da7a-ba55-da7aba5e57ab:1-444:1000041-1000503:2000041-
LAST_CONFLICT_FREE_TRANSACTION:
da7aba5e-dead-da7a-ba55-da7aba5e57ab:444
40 / 65

Group Replication Status
mysql> select * from replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
GROUP_NAME: da7aba5e-dead-da7a-ba55-da7aba5e57ab
SOURCE_UUID: da7aba5e-dead-da7a-ba55-da7aba5e57ab
THREAD_ID: NULL
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET:
72149827-e1cc-11e6-9daf-08002789cd2e:1,
740e1fd2-e1cc-11e6-a8ec-08002789cd2e:1-2,
74dc6ab2-e1cc-11e6-92aa-08002789cd2e:1-2,
da7aba5e-dead-da7a-ba55-da7aba5e57ab:1-444:1000041-1000503:2000041-2000648
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
41 / 65

Group Replication Read or Write.
mysql> select @@global.super_read_only;
+--------------------------+
| @@global.super_read_only |
+--------------------------+
| 1 |
+--------------------------+
43 / 65

Group Replication Lag
SELECT sys.gtid_count(
GTID_SUBTRACT(
(
SELECT
Received_transaction_set
FROM performance_schema.replication_connection_status
WHERE
Channel_name = 'group_replication_applier'
),
(SELECT
@@global.GTID_EXECUTED)
)
)
)
Thanks to @lefred:
https://github.com/lefred/mysql_gr_routing_check/
44 / 65

Commands
mysql> SHOW SLAVE STATUS FOR CHANNEL 'group_replication_recovery'G
mysql> SHOW SLAVE STATUS FOR CHANNEL 'group_replication_applier'G
ERROR 3139 (HY000): SHOW SLAVE STATUS cannot be performed on channel 'group_re
45 / 65

Member State Changes
Can use improvements:
#84798: Group Replication can use some verbosity in the error
log
46 / 65

Multi Node Conﬂicts
Optimistic Locking
First committer wins
#84730: ability to troubleshoot transaction rollbacks
ERROR 3101 (HY000) at line 1: Plugin instructed the server
to rollback the current transaction.
47 / 65

Backups
48 / 65

Backups
It's just InnoDB, use your favorite GTID supported backup tool:
Percona XtraBackup
MySQL Enterprise Backup
mysqldump
mysqlpump
mydumper
but...
49 / 65

Backups
It's just InnoDB, use your favorite GTID supported backup tool:
Percona XtraBackup
MySQL Enterprise Backup
mysqldump
mysqlpump
mydumper
but...
#84799: mysqldump --single-transaction uses savepoints, does
not work with GR
mydumper --use-savepoints is also affected
50 / 65

Load Balancers
51 / 65

MySQL Router (Beta)
MySQL Router is part of MySQL InnoDB Cluster
52 / 65

MySQL Router (Beta)
I brieﬂy evaluated, but quickly ran into serious problems:
missing a lot of features
pretty unknown in the community
...:
#83237: mysqlrouter connects to wrong metadata server (a
partitioned node)
Visibility:
#83236: How to see mysqlrouter membership status?
53 / 65

MySQL Router (Beta)
I brieﬂy evaluated, but quickly ran into serious problems:
missing a lot of features
pretty unknown in the community
...:
#83237: mysqlrouter connects to wrong metadata server (a
partitioned node)
Visibility:
#83236: How to see mysqlrouter membership status?
I might consider to re-evaluate, but ﬁrst ^^
54 / 65

ProxySQL
Really Open Source!
Becoming very popular
Example Implementation http://lefred.be/content/ha-with-
mysql-group-replication-and-proxysql/
Careful: #2: using multiple hostgroups/schedulers with
proxysql_groupreplication_checker.sh can
cause unwanted state changes
55 / 65

Improvements
56 / 65

#84784 - Nodes Do Not Reconnect
Nodes do not reconnect to the group replication once they got
disconnected, causing nodes to drop from the cluster and can
lead to losing the whole cluster availability
57 / 65

Improvements
Reduce impact on applications:
#84731: mysql client connections get stuck during GR start
59 / 65

Improvements
Reduce impact on applications:
#84731: mysql client connections get stuck during GR start
Partition Tolerance issues, split brain cannot be prevented:
#84727: partitioned nodes still accept writes: queries hang
#84729: block reads on partitioned nodes *
#84784: Nodes Do Not Reconnect
#84795: STOP GROUP_REPLICATION sets
super_read_only=off
60 / 65

Improvements
Stability:
#84785: Prevent Large Transactions in Group Replication
#84792: Member using 100% CPU in idle cluster
61 / 65

Improvements
Stability:
#84785: Prevent Large Transactions in Group Replication
#84792: Member using 100% CPU in idle cluster
Usability:
#84794: cannot kill query that is stuck inside GR
#84799: mysqldump --single-transaction uses
savepoints, does not work with GR
#84798: Group Replication can use some
verbosity in the error log
62 / 65

For Every Bug Fixed: 1 Beer
63 / 65

Summary
64 / 65

Summary
Use Cases for Group Replication
Environments with strict durability requirements
Ensure split-brain can be completely avoided
Write to multiple nodes
('scalability' by splitting write/read workloads)
Not recommended (yet)
Improve failover time
Reducing impact on applications can be improved
...
65 / 65

MySQL Group Replication

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to MySQL Group Replication (20)

More from Kenny Gryp (10)

Recently uploaded (20)

MySQL Group Replication