Why MySQL Replication Fails, and How to Get it Back

Why MySQL Replication Fails,
and How to Get it Back
September, 26, 2017
Sveta Smirnova

∙ MySQL Support engineer
∙ Author of
∙ MySQL Troubleshooting
∙ JSON UDF functions
∙ FILTER clause for MySQL
∙ Speaker
∙ Percona Live, OOW, Fosdem,
DevConf, HighLoad...
Sveta Smirnova
2

∙ Exists since very ﬁrst versions
Replication in MySQL
4

∙ Easy to use
4

∙ Easy to use
∙ Minimal setup
4

∙ Master
∙ --log-bin
∙ --server-id
∙ GRANT REPLICATION SLAVE ON *.* ...
Turn Replication On
5

∙ Master
∙ --log-bin
∙ --server-id
∙ GRANT REPLICATION SLAVE ON *.* ...
∙ Slave
∙ --server-id
∙ CHANGE MASTER ...
∙ START SLAVE
Turn Replication On
5

Increased Resource Usage on Master
13

Master Slave
← Initiates
Asynchronous
16

Master Slave
← Initiates
← Requests a packet
Asynchronous
16

Master
Sends the packet →
Slave
← Initiates
Asynchronous
16

Master
Slave
← Initiates
... ?
Asynchronous
16

∙ Network error
∙ Authorization
Did Slave Recieve Data?
17

Slave_IO_State: Waiting for master to send event
Master_Host: 127.0.0.1
Master_User: root
Master_Port: 13000
Connect_Retry: 60
Master_Log_File: master-bin.000002
Read_Master_Log_Pos: 63810611
Relay_Log_File: slave-relay-bin-master@002d1.000004
Relay_Log_Pos: 1156
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1032
Last_Error: Could not execute Update_rows event on...
Skip_Counter: 0
Exec_Master_Log_Pos: 989
Relay_Log_Space: 63814652
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No | Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Update_rows event on...
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_UUID: d08c509e-6857-11e6-8872-30b5c2208a0f
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 10
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 160823 15:11:21
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name: master-1 | Master_TLS_Version:
∙ IO thread Conﬁguration
∙ SQL thread Conﬁguration
∙ IO thread State
∙ SQL thread State
∙ Errors
∙ Only last one
∙ All are in the error log
Tool #1: SHOW SLAVE STATUS
18

Slave_IO_Running: Connecting
Slave_SQL_Running: Yes
...
Last_IO_Errno: 1045
Last_IO_Error: error connecting to master ’root@127.0.0.1:13000’ -
Last_SQL_Errno: 0
Last_SQL_Error:
...
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp: 160824 03:18:36
Last_SQL_Error_Timestamp:
Network Errors
19

mysql> select * from performance_schema.replication_connection_statusG
*************************** 1. row ***************************
CHANNEL_NAME:
GROUP_NAME:
SOURCE_UUID:
THREAD_ID: NULL
SERVICE_STATE: CONNECTING
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET:
LAST_ERROR_NUMBER: 1045
LAST_ERROR_MESSAGE: error connecting to master ’root@127.0.0.1:13000’ -
retry-time: 60 retries: 4
LAST_ERROR_TIMESTAMP: 2016-08-24 03:21:36
1 row in set (0,01 sec)
#2: connection status in Performance Schema
20

2016-08-24T00:18:36.077384Z 3 [ERROR] Slave I/O for channel ’’: error connecting to
master ’root@127.0.0.1:13000’ - retry-time: 60 retries: 1, Error_code: 1045
#3: Error Log File
21

$ perror 1045
MySQL error code 1045 (ER_ACCESS_DENIED_ERROR): Access denied for user ’%-.48s’@’%-.64s’
(using password: %s)
#4: perror
22

∙ On the slave
$ mysql -h127.0.0.1 -P13000 -uslave_user -pslave_password
Warning: Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user ’slave_user’@’localhost’ (using password: YES
#5: MySQL Command Line Client
23

∙ On the slave
∙ On the master
mysql> SHOW GRANTS;
+-----------------------------------------+
| Grants for slave_user@% |
+-----------------------------------------+
| GRANT SELECT ON *.* TO ’slave_user’@’%’ |
+-----------------------------------------+
1 row in set (0.00 sec)
23

∙ On the slave
∙ On the master
∙ Fix privileges on master
GRANT REPLICATION SLAVE
ON *.* TO ’slave user’@’%’
23

∙ On the slave
∙ On the master
∙ Fix privileges on master
∙ Restart replication
23

Master Slave
← Initiates
Semisynchrous plugin
24

Master Slave
← Initiates
24

Master
Slave
← Initiates
24

Master
Waits ”Ack”
Slave
← Initiates
24

Master
Waits ”Ack”
Slave
← Initiates
← Sends ”Ack”
24

∙ Writes on master are slower
Semisynchrous: Troubleshooting
25

∙ How many ”Ack”-s master waits?
25

∙ Before 5.7: from single slave
25

∙ Now in MySQL:
rpl semi sync master wait for slave count
25

∙ Now in MySQL:
rpl semi sync master wait for slave count
∙ Won’t wait others
25

∙ What happens in case of timeout?
25

∙ Replication becomes asynchronous
25

∙ What does ”Ack” mean?
25

∙ Event written into relay log
25

∙ Event written into relay log
∙ It is unknown if event applied
25

IO thread
Reads from the master
SQL thread
Two Kinds of Slave Threads
26

IO thread
Stores in the relay log
SQL thread
26

IO thread
SQL thread
← Reads from relay log
26

IO thread
SQL thread
← Reads from relay log
Executes
26

∙ Easier for troubleshooting
Single SQL Thread
27

∙ Easier for troubleshooting
∙ Slower than master
∙ High parallel load
Single SQL Thread
27

Writes on the Slave: Single SQL Thread
29

∙ MySQL: --slave parallel workers
∙ MySQL: --slave parallel type=DATABASE | LOGICAL CLOCK
∙ MySQL 8.0.1+:
SET @@GLOBAL.binlog transaction dependency tracking =
WRITESET | WRITESET SESSION | COMMIT ORDER;
Performance Tuning
31

∙ MariaDB: --slave parallel threads
∙ MariaDB: --slave parallel max queued
∙ MariaDB: --slave domain parallel threads
∙ MariaDB: --slave parallel mode=optimistic | conservative |
aggressive | minimal | none
Performance Tuning
31

mysql> select WORKER_ID, SERVICE_STATE, LAST_SEEN_TRANSACTION, LAST_ERROR_NUMBER,
-> LAST_ERROR_MESSAGE from performance_schema.replication_applier_status_by_workerG
*************************** 1. row ***************************
WORKER_ID: 1
SERVICE_STATE: OFF
LAST_SEEN_TRANSACTION: d318bc17-66dc-11e6-a471-30b5c2208a0f:4988
LAST_ERROR_MESSAGE:
*************************** 2. row ***************************
WORKER_ID: 3
SERVICE_STATE: OFF
LAST_SEEN_TRANSACTION: d318bc17-66dc-11e6-a471-30b5c2208a0f:4986
LAST_ERROR_MESSAGE: Worker 2 failed executing transaction...
#6: Error of One Thread Stops All
32

MariaDB [test]> select id, command, time, state from information_schema.processlist
-> where user=’system user’;
+----+---------+------+------------------------------------------------------------------+
| id | command | time | state |
+----+---------+------+------------------------------------------------------------------+
| 25 | Connect | 4738 | Waiting for master to send event |
| 24 | Connect | 5096 | Slave has read all relay log; waiting for the slave I/O thread t |
| 23 | Connect | 0 | Waiting for work from SQL thread |
| 22 | Connect | 0 | Unlocking tables |
| 21 | Connect | 0 | Update_rows_log_event::ha_update_row(-1) |
| 20 | Connect | 0 | Waiting for prior transaction to start commit before starting ne |
| 17 | Connect | 0 | Update_rows_log_event::find_row(-1)
...
#6: Error of One Thread Stops All
32

∙ Different data
∙ Slave cannot apply event from relay log
Which Kind of Errors?
33

∙ Different data
∙ Slave cannot apply event from relay log
∙ Different errors on master and slave
∙ Triggers
∙ Transactional and non-transactional tables in
the same transaction
Which Kind of Errors?
33

∙ Did table change outside of the replication?
∙ How?
∙ Can it cause conﬂict with changes on the
master?
Different Data on Master and Slave
34

∙ Are table structures identical?
∙ Percona Toolkit
pt-table-checksum, pt-table-sync
∙ MySQL Utilities
mysqlrplsync, mysqldbcompare, mysqldiff
34

∙ Are table structures identical?
∙ Are changes in the correct order?
∙ mysqlbinlog
∙ Application logic on the master
34

Master
Recieves a change
Storage Engine
Logical
35

Master
Recieves a change
Sends to SE →
Storage Engine
Logical
35

Master
Recieves a change
Sends to SE →
Storage Engine
Writes into table
Logical
35

Master
Recieves a change
Sends to SE →
Storage Engine
Writes into table
← Returns control
Logical
35

Master
Recieves a change
Sends to SE →
Writes into binary log
Storage Engine
Writes into table
← Returns control
Logical
35

Master
Recieves a change
Sends to SE →
Writes into binary log
Synchronizes →
Storage Engine
Writes into table
← Returns control
← Synchronizes
Logical
35

∙ More writes
∙ RBR: --binlog row image
Master Performance
36

∙ More writes
∙ --binlog cache size
Watch Binlog cache disk use
Master Performance
36

∙ More writes
∙ --binlog cache size
Watch Binlog cache disk use
∙ --binlog stmt cache size
Watch Binlog stmt cache disk use
Master Performance
36

∙ More writes
∙ Synchronization
∙ --binlog sync
∙ Do not disable!
∙ You may set it greater than 1
Master Performance
36

∙ Binary log lifetime
∙ --expire log days
Master Behavior
37

∙ Synchronization
∙ SBR is not safe with READ COMMITTED and READ
UNCOMMITTED
Master Behavior
37

∙ Synchronization
∙ Order of records in the binary log
∙ Non-deterministic events and SBR
Master Behavior
37

Client Binary log
Statement-Based Binary Log Format
38

Client
INSERT INTO ... →
Binary log
38

Client
INSERT INTO ... →
Binary log
SET TIMESTAMP...
38

Client
INSERT INTO ... →
Binary log
SET TIMESTAMP...
SET sql mode...
38

Client
INSERT INTO ... →
Binary log
SET TIMESTAMP...
SET sql mode...
INSERT INTO ...
38

SBR: Strong Sides
39

∙ Table deﬁnitions on master and slave can
signiﬁcantly vary
SBR: Strong Sides
39

signiﬁcantly vary
∙ Usually less writes
∙ There are exceptions!
SBR: Strong Sides
39

signiﬁcantly vary
∙ Human readable
SBR: Strong Sides
39

signiﬁcantly vary
∙ Human readable
∙ Easy to troubleshoot
SBR: Strong Sides
39

mysql> SHOW BINLOG EVENTS IN ’mysql-bin.000316’ FROM 422;
+------------------+-----+------------+------------+-------------+---------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+-----+------------+------------+-------------+---------------------------------+
| mysql-bin.000316 | 422 | Query | 1456667904 | 509 | BEGIN |
| mysql-bin.000316 | 509 | Query | 1456667904 | 609 | use ‘PgDay‘; update ai set f1=1 |
| mysql-bin.000316 | 609 | Xid | 1456667904 | 640 | COMMIT /* xid=60328 */ |
+------------------+-----+------------+------------+-------------+---------------------------------+
3 rows in set (0,12 sec)
#7: SHOW BINLOG EVENTS
40

∙ Not all queries are safe
∙ Non-deterministic functions
∙ MySQL extentions
∙ Triggers
∙ Mix with non-transactional tables
∙ Temporary tables
SBR: Weak Sides
41

∙ Order of events matter!
∙ Row-based locks
SBR: Weak Sides
41

∙ Order of events matter!
∙ Row-based locks
∙ Triggers
SET GLOBAL slave skip counter – No GTIDs!
Skip transaction – GTIDs
Synchronize tables!
SBR: Weak Sides
41

Client Binary log
Row-Based Binary Log Format
42

Client
UPDATE ... →
Binary log
42

Client
UPDATE ... →
Binary log
SET TIMESTAMP...
42

Client
UPDATE ... →
Binary log
SET TIMESTAMP...
SET sql mode...
42

Client
UPDATE ... →
Binary log
SET TIMESTAMP...
SET sql mode...
Row before changes
42

Client
UPDATE ... →
Binary log
SET TIMESTAMP...
SET sql mode...
Row before changes
Row with changes
42

∙ Safe
∙ You do not need to care about
Order of events
Triggers
Functions
Which queries you send to master
RBR: Strong Sides
43

∙ Sensitive for table structures
∙ More writes
∙ --binlog row image=FULL | MINIMAL | NOBLOB
∙ Harder to read
RBR: Weak Sidex
44

$ mysqlbinlog ./mysqld.1/data/master-bin.000001 --start-position=989 --stop-position=1213
...
# at 1167
#160822 14:15:11 server id 1 end_log_pos 1213 CRC32 0x1f346c6b
Update_rows: table id 109 flags: STMT_END_F
BINLOG ’
v966VxMBAAAAKwAAAI8EAAAAAG0AAAAAAAEAAm0yAAJ0MQABAwABY2HOoQ==
v966Vx8BAAAALgAAAL0EAAAAAG0AAAAAAAEAAgAB///+BQAAAP4GAAAAa2w0Hw==
’/*!*/;
ROLLBACK /* added by mysqlbinlog */ /*!*/;
SET @@SESSION.GTID_NEXT= ’AUTOMATIC’ /* added by mysqlbinlog */ /*!*/;
...
#8: mysqlbinlog
45

$ mysqlbinlog -v ./mysqld.1/data/master-bin.000001 --start-position=989 --stop-position=1213
...
# at 1167
#160822 14:15:11 server id 1 end_log_pos 1213 CRC32 0x1f346c6b
Update_rows: table id 109 flags: STMT_END_F
BINLOG ’
v966VxMBAAAAKwAAAI8EAAAAAG0AAAAAAAEAAm0yAAJ0MQABAwABY2HOoQ==
v966Vx8BAAAALgAAAL0EAAAAAG0AAAAAAAEAAgAB///+BQAAAP4GAAAAa2w0Hw==
’/*!*/;
### UPDATE ‘m2‘.‘t1‘
### WHERE
### @1=5
### SET
### @1=6
ROLLBACK /* added by mysqlbinlog */ /*!*/;
SET @@SESSION.GTID_NEXT= ’AUTOMATIC’ /* added by mysqlbinlog */ /*!*/;
#8: mysqlbinlog
46

∙ You must specify
∙ Name of the master’s binary log ﬁle
∙ Position
Position-Based
47

∙ Position
∙ From the troubleshooting point of view
∙ Event executes if on the current position
Position-Based
47

∙ Position
∙ Easy to skip
Position-Based
47

∙ Position
∙ Easy to skip
∙ Easy to move position backward
Position-Based
47

∙ Position
∙ Easy to skip
∙ Easy to move position backward
∙ No conﬂict resolution
Position-Based
47

∙ Each transaction has unique number: GTID
Global Transaction Identiﬁers (GTID)
48

∙ MySQL: AUTO POSITION=1
48

∙ MariaDB: master use gtid = { slave pos | current pos }
48

∙ No need to specify binary log and position
48

∙ No need to specify binary log and position
∙ Hard to skip erroneous event
48

sveta@thinkie> mysqlslavetrx --gtid-set=fb776095-8474-11e5-ad41-30b5c2208a0f:3
--slaves=root:@127.0.0.1:13001
WARNING: Using a password on the command line interface can be insecure.
#
# GTID set to be skipped for each server:
# - 127.0.0.1@13001: fb776095-8474-11e5-ad41-30b5c2208a0f:3
#
# Injecting empty transactions for ’127.0.0.1:13001’...
#
#...done.
#
#9: mysqlslavetrx
49

∙ Same speciﬁcs and methods
Complicated Setups
51

∙ Multiply complexity on number of channels
Complicated Setups
51

∙ Multiply complexity on number of channels
∙ Control writes
∙ What
∙ On which server
Complicated Setups
51

Master
∙ Binary log
∙ Binlog Dump
Slave
∙ Relay log
∙ IO thread
∙ SQL thread
Master and Slave
52

Slave 1
∙ Relay log
∙ IO thread
∙ SQL thread
Master
∙ Binary log
∙ Binlog Dump
∙ Binlog Dump
Slave 2
∙ Relay log
∙ IO thread
∙ SQL thread
Master and Two Slaves
53

Master 1
∙ Binary log
∙ Binlog Dump
∙ Relay log
∙ IO thread
∙ SQL thread
Master 2
∙ Relay log
∙ IO thread
∙ SQL thread
∙ Binary log
∙ Binlog Dump
Circular
54

Master 1
∙ Binary log
∙ Binlog Dump
Slave
∙ Relay log
∙ IO thread
∙ SQL thread
∙ Relay log
∙ IO thread
∙ SQL thread
Master 2
∙ Binary log
∙ Binlog Dump
Multiple Masters (Multi-channel): 5.7+
55

∙ Multiple sets of relay logs
∙ Multiple IO threads
∙ Multiple SQL threads
∙ MySQL: --slave parallel workers for each
Multi-Master: Troubleshooting
56

∙ Independent channels
56

∙ Error in one stops only one
56

∙ Error in one stops only one
∙ No automatic conﬂict resolution
56

∙ Error log ﬁle
Basic Tools
58

∙ Error log ﬁle
∙ On the slave
∙ SHOW SLAVE STATUS
∙ MySQL: Tables in Performance Schema
∙ System database mysql
Basic Tools
58

∙ Error log ﬁle
∙ On the slave
∙ On the master
∙ SHOW MASTER STATUS
∙ SHOW BINLOG EVENTS
∙ mysqlbinlog
Basic Tools
58

∙ Error log ﬁle
∙ On the slave
∙ On the master
∙ Percona Toolkit
Basic Tools
58

∙ Error log ﬁle
∙ On the slave
∙ On the master
∙ Percona Toolkit
∙ MySQL Utilities
Basic Tools
58

∙ Always available, requires setup
∙ Asynchronous
∙ Master
∙ Keeps all changes in the binary log
Two formats: ROW and STATEMENT
∙ Slave
∙ IO thread reads from the master into relay log
∙ SQL thread executes updates
Multiple SQL threads in 5.6+
Multiple channels/sources (masters) in 5.7+
∙ GTID in 5.6+
Replication Must Know
59

∙ Master
∙ Same as for standalone server
∙ More writes and consistency checks
Typical Issues
60

∙ Master
∙ Slave IO thread
∙ Common network issues
∙ mysql command line client for tests
Typical Issues
60

∙ Master
∙ Slave IO thread
∙ Slave SQL thread
∙ Regular query-related issues
∙ Regular storage engine issues
∙ Less execution threads than on master
Typical Issues
60

∙ Basic Techniques – troubleshooting
webinar
∙ Troubleshooting hardware resource usage
∙ Introduction into storage engine
troubleshoot...
∙ Percona Toolkit
∙ MySQL Utilities
∙ Book MySQL High Availability
∙ MySQL Replication Team blog
More Information
61

http://www.slideshare.net/SvetaSmirnova
https://twitter.com/svetsmirnova
https://github.com/svetasmirnova
Contact Information
62

∙ ... Case Study: .IE Continuous Restore ...
Marcelo Altmann - Percona, Mick Begley - IE Domain Registry
Tuesday 2:20PM-3:10PM @ Goldsmith 3
∙ A ... Walkthrough on pt-stalk
Marcelo Altmann - Percona, Marcos Albe - Percona
Wednesday 3:20PM-4:10PM @ Field Suite 2
∙ Percona XtraBackup Best Practices
Marcelo Altmann - Percona
Wednesday 4:30PM-4:55PM @ Sky Suite 2
∙ MySQL-MongoDB-Redis-Cassandra-HBase
Marcos Albe - Percona
Wednesday 4.55PM-5.20PM @ Sky Suite 2
Support Team at Percona Live
63

Why MySQL Replication Fails, and How to Get it Back

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Why MySQL Replication Fails, and How to Get it Back (20)

More from Sveta Smirnova (20)

Recently uploaded (20)

Why MySQL Replication Fails, and How to Get it Back