Keith Larson Replication

http://hiconsumption.com/2012/07/star-wars-recreations-of-iconic-photographs-by-david-eger/

<Insert Picture Here>

MySQL: Replication
Keith Larson
keith.larson@oracle.com
MySQL Community Manager

Safe Harbor Statement

The following is intended to outline our general product
direction. It is intended for information purposes only, and may
not be incorporated into any contract. It is not a commitment to
deliver any material, code, or functionality, and should not be
relied upon in making purchasing decisions.

The development, release, and timing of any features or
functionality described for Oracle’s products remains at the
sole discretion of Oracle.

http://jennysung.com/wp-content/uploads/2012/03/stormtroopers1.jpg

TM & © 2012 Lucasfilm Ltd. All rights reserved. ©2012 The LEGO Group. All rights reserved.

Who am I and who are you?
Who are you?
– DBAs?
– Developers?
• PHP
• Perl
• Python
• .net
– Admins?

.

http://3.bp.blogspot.com/-IXu-9-s-8Fg/TvTs8JbgvdI/AAAAAAAAJpo/at-pjGsDlXg/s1600/Bat%2BStorm%2BTrooper%2BLego.jpg

Copyright Oracle Corporation 2012 4

Session Agenda


MySQL Replication Overview

Replication Configuration

Examples of a real world set up

MySQL 5.6 Replication Features

Monitoring – MySQL Enterprise Monitor

Overview Configuration Features Monitor

Copyright Oracle Corporation 2012


http://farm3.static.flickr.com/2646/3678467304_66908d66d4.jpg



Duplicates database from a
“master” to a “slave”
Redundant copies of the data
provide foundation for High
Availability

Scale out by distributing queries
across the replication cluster

http://thewondrous.com/wp-content/uploads/2010/04/Largest-display-of-Star-Wars-clone-troopers-built-with-interlocking-plastic-bricks.jpg


Who Uses Replication ?

“In my opinion, MySQL is the only database
we would ever trust to power the Zappos.com "On any given day we can sell close to
website.” 300,000 tickets on the Web site using
MySQL as the database to search for
events. It is amazing."

“As a leader in our field, we are committed
to providing the best service to our users,
and a web experience that meets members
expectations and that starts with IT”
“We are one of the largest MySQL
web sites in production

“They have a master server for all writes and slave
"craigslist infrastructure servers for most Reads. The secret truth they claim
could not have handled the behind configuring the master and slave machines
exponential growth in traffic is to make sure the slave machines are faster than
without MySQL.” the masters”


Who Uses Replication ?




Native in MySQL

Replicate within and across data centers

Failover is either scripted or provided by additional middleware

Supports Asynchronous (standard) and Semi-Synchronous replication

Each slave adds minimal load on master

http://www.betabeat.com/2011/09/02/clone-wars-rise-of-the-fast-follower-startups/



Replication Topologies




Used for Scalability and HA

Asynchronous as standard

Semi-Synchronous support added in MySQL 5.5


Replication Threads

Binlog dump thread

Slave I/O thread

Slave SQL thread

Replication Files

relay log

master info log

relay log info log



http://technabob.com/blog/2011/12/16/lego-stormtroope-american-beauty/








The master creates a thread to send the binary log contents to a
Replication Threads slave when the slave connects. This thread can be identified in the
output of SHOW PROCESSLIST on the master as the Binlog Dump

Binlog dump thread thread.

Slave I/O thread

Slave SQL thread

The binlog dump thread acquires a lock on the master's binary
log for reading each event that is to be sent to the slave. As soon
as the event has been read, the lock is released, even before the
Replication Files event is sent to the slave.

relay log *************************** 2. row ***************************
Id: 27

master info log User: replication

relay log info log Host: 192.168.0.11:47129
db: NULL
Command: Binlog Dump
Time: 499
State: Master has sent all binlog to slave; waiting for binlog to
be updated
Info: NULL








When a START SLAVE statement is issued on a slave server, the
Replication Threads slave creates an I/O thread, which connects to the master and

Binlog dump thread asks it to send the updates recorded in its binary logs.

Slave I/O thread The slave I/O thread reads the updates that the master's Binlog

Slave SQL thread Dump thread sends and copies them to local files that comprise
the slave's relay log.
Replication Files The state of this thread is shown as Slave_IO_running in the

relay log output of SHOW SLAVE STATUS or as Slave_running in the

master info log output of SHOW STATUS.

relay log info log Slave_IO_Running: Yes
| Slave_running | ON |








Replication Threads

Binlog dump thread

Slave I/O thread The slave creates an SQL thread to read the relay log

Slave SQL thread that is written by the slave I/O thread and execute the
events contained therein.
Replication Files

relay log Slave_SQL_Running: Yes

master info log

relay log info log








Replication Threads

Binlog dump thread

Slave I/O thread

Slave SQL thread
The relay log consists of the events read from the binary
Replication Files log of the master and written by the slave I/O thread.

relay log Events in the relay log are executed on the slave as part

master info log of the SQL thread. caption text here.

relay log info log








Replication Threads

Binlog dump thread

Slave I/O thread

Slave SQL thread
The master info log contains status and current
Replication Files configuration information for the slave's connection to

relay log the master. This log holds information on the master

master info log host name, login credentials, and coordinates indicating

relay log info log how far the slave has read from the master's binary log.








Replication Threads

Binlog dump thread

Slave I/O thread

Slave SQL thread

Replication Files

relay log

master info log The relay log info log holds status information about the

relay log info log execution point within the slave's relay log



Replication Formats

Statement-based replication (SBR)
– Advantages
• Less data written to log files
• taking and restoring from backups are faster
– Disadvantages
• Some functions or queries are
nondeterministic and hard to replicate


Row-based replication (RBR)
– Advantages
• safest form of replication
• Fewer row locks are required on the slave
– Disadvantages
• generate more data that must be logged*
• Blob values take longer to replicate


Mixed-format
– statement-based logging is used by default
– automatically switches to row-based logging in
particular cases
http://dev.mysql.com/doc/refman/5.5/en/replication-formats.html
http://dev.mysql.com/doc/refman/5.1/en/binary-log-setting.html


Asynchronous is standard

events are occurring independently


The master writes events to its binary log but does not know whether
or when a slave has retrieved and processed them


if the master crashes, transactions that it has committed might not
have been transmitted to any slave

●
Original patch: Mark Callaghan and Wei Li, Google
Adoptions: Zhenxing He, Sun Microsystems
●
acknowledges receipt of a transaction's events only after the events
have been written to its relay log and flushed to disk

●
best for close servers communicating over fast networks

●
timeout occurs without any slave having acknowledged the
transaction, the master reverts to asynchronous replication. When at
least one semisynchronous slave catches up, the master returns to
semisynchronous replication.
http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html


MySQL Replication Configuration

http://hiconsumption.com/2012/07/star-wars-recreations-of-iconic-photographs-by-david-eger/


Master Setup:
# vi /etc/my.cnf
[mysqld]
server-id=1
log-bin = /var/lib/mysql/yoda-bin

Start and Log into MySQL
master_yoda>show master statusG
*************************** 1. row ***************************
File: yoda-bin.000001
Position: 114
Binlog_Do_DB:
Binlog_Ignore_DB:
1 row in set (0.00 sec)

# mysqldump -p --all-databases --master-data=2 > /tmp/replication_example.sql
THIS LOCKS THE DATABASE!

http://www.minifigures.co.uk/lego/star-wars/master-yoda/
http://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html



On Master :

mysql_yoda>CREATE USER 'replication'@'192.168.0.%' IDENTIFIED BY
'slavepass';
mysql_yoda>GRANT REPLICATION SLAVE ON *.* TO 'replication'@'%';
mysql_yoda>flush privileges;

Adjust all firewall rules if required for MySQL Port. (3306)

http://dev.mysql.com/doc/refman/5.5/en/replication-howto-masterbaseconfig.html


Slave Setup:
vi /etc/my.cnf
[mysqld]
Server-id=2
relay-log=/var/lib/mysql/luke-relay-bin

# optional below
log-bin = /var/lib/mysql/luke-bin

luke>show master statusG
*************************** 1. row ***************************
File: luke-bin.000003
Position: 114
Binlog_Do_DB:
Binlog_Ignore_DB:

LOAD DATA:
# mysql --user=root -p < /tmp/replication_example.sql

http://www.minifigures.co.uk/lego/star-wars/luke-skywalker-on-tatooine/


On Slave :

mysql_luke> CHANGE MASTER TO
MASTER_HOST='yoda',
MASTER_USER='replication',
MASTER_PASSWORD='slavepass',
MASTER_PORT=3306,
MASTER_LOG_FILE='yoda-bin.000002',
MASTER_LOG_POS=83415,
MASTER_CONNECT_RETRY=10;

We gathered this info from the mysqldump file via the
“ --master-data=2 ” flag.

-- CHANGE MASTER TO
MASTER_LOG_FILE='yoda-bin.000002',
MASTER_LOG_POS=83415;

mysql_luke > start slave;
Query OK, 0 rows affected (0.00 sec)

On Slave :
mysql_luke> show slave statusG

*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: yoda
Master_User: replication
Master_Port: 3306
Connect_Retry: 10
Master_Log_File: yoda-bin.000003
Read_Master_Log_Pos: 323
Relay_Log_File: luke-relay-bin.000004
Relay_Log_Pos: 475
Relay_Master_Log_File: yoda-bin.000003
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Master_Server_Id: 1
Master_UUID: 75d407df-2be4-11e1-9668-b4be9bce39b0
Seconds_Behind_Master: 0
SQL_Remaining_Delay: NULL


Master to Master
mysql_yoda>CHANGE MASTER TO
mysql> SET GLOBAL auto_increment_offset=2;
MASTER_HOST='luke',
mysql> SET GLOBAL auto_increment_increment=2;
MASTER_USER='replication2',
MASTER_PASSWORD='slavepass',
MASTER_PORT=3306,
MASTER_LOG_FILE='luke-bin.000005',
mysql_yoda>show global variables like '%auto_increment%';
MASTER_LOG_POS=295,
+--------------------------+-------+
MASTER_CONNECT_RETRY=10;
| Variable_name | Value |
mysql_yoda> start slave;
+--------------------------+-------+
| auto_increment_increment | 2 |
| auto_increment_offset | 1 |
+--------------------------+-------+

mysql_luke>show global variables like '%auto_increment%';
+--------------------------+-------+
+--------------------------+-------+
| auto_increment_increment | 2 |
| auto_increment_offset | 2 |
+--------------------------+-------+

mysql_luke>SHOW MASTER STATUSG
*************************** 1. row ***************************
File: luke-bin.000005
Position: 295

http://legostarwars.wikia.com/wiki/Luke_Skywalker_%28Jedi%29


MySQL 5.6: Replication Utilities (Python)
Setting up replication with utilities
mysqlreplicate --master=root@master.example.com
--slave=root@slave.example.com --rpl-user=repl:xyzzy

http://mysqlworkbench.org/utilities/man/mysqlreplicate.html

Require master and slave to be off-line!

https://github.com/greyrl/generaltools/blob/master/mysqlreplicate.py


MySQL 5.6: Replication
New!

http://www.flickr.com/photos/kalexanderson/7719347704/in/set-72157628651430439/


New!


Global Transaction IDs

MySQL Utilities

Multi-Threaded Slaves

Binary Log Group Commit

Optimized Row Based Replication

Crash Safe Slaves

Replication Event Checksums

Time Delayed Replication

http://dev.mysql.com/doc/refman/5.6/en/replication-features.html
http://dev.mysql.com/downloads/mysql/#downloads Under Development Releases


New! Global Transaction Ids
 Simple to track & compare replication across the cluster
- Unique identifier for each transaction written to the Binlog
 Automatically identify the most up-to-date slave for failover
 Deploy n-tier replication hierarchies

Master
GTID=123456

GTID=123456

GTID=123456 GTID=123456


New! Global Transaction Ids
Slave(s) can start from same GTID
●
reference a global transaction
identifier and not have to convert
binary log filenames and offsets
between different servers
Master

GTID=123456

CHANGE MASTER TO??

GTID=123456

GTID=123456 GTID=123456


MySQL 5.6: Replication HA Utilities (Python)
 Enabling self-healing replication topologies
 Automated failover & recovery
- mysqlfailover Utility
Failed
 Switchover & administration
Master
- mysqlrpladmin Utility

Monitoring HA Utilities
Promoted
Master  Delivers HA within the core MySQL
distribution
- Eliminates the need for 3rd party solutions
- Allows extensibility to support variety of
HA mechanisms
https://launchpad.net/mysql-utilities/

Slaves

• Database Operations
• mysqldbcompare – compare databases
• mysqldbcopy – copy databases between servers
• mysqldbexport – export metadata and data
• mysqldbimport – import metadata and data
• mysqldiff – compare object definitions
• General Utilities
• mysqldiskusage – show disk usage for databases
• mysqlindexcheck – check for redundant indexes
• mysqlmetagrep – search metadata
• mysqlprocgrep – search process information
• mysqluserclone – clone a user account

• High Availability
• mysqlfailover – automatic failover for MySQL 5.6.5+
• mysqlreplicate – setup replication
• mysqlrpladmin – general replication administration
• switchover
• failover for MySQL 5.6.5
• mysqlrplcheck – check replication configuration

• mysqlrplshow – display map of replication topology
• Server Operations
• mysqlserverclone – start a scratch server
• mysqlserverinfo – show server information

How do I access the utilities?

 Increases slave throughput,
Multi-Threaded Slave Performance reducing lag
300  All transactions received into
250
slave’s relay log
 Implements multiple SQL threads,
Queries per Second

200
based on database
150
 Applies events to different
100
databases in parallel
50
 Great for systems which isolate
0
0 5 10
application data using databases
Worker Threads
– e.g. multi-tenant

 5x performance gain
 SysBench, running across 10 x schemas
 Oracle Linux 6.1, Oracle Sun Fire x4150 m2 Server



Throughput of slave increased by allowing multiple slave threads:
●
0 - functionality disabled
●
0 to 1024
Exec_Master_Log_Posn in SHOW SLAVE STATUS represents a
“low-water” mark, before which no uncommitted transactions remain.
Configure using:
●
slave-parallel-workers=4
On a per-database basis
●
can process successive transactions on a given database
without waiting for updates on other databases to complete

http://dev.mysql.com/doc/refman/5.6/en/replication-options-slave.html#sysvar_slave_parallel_workers


*************************** 1. row ***************************
....
Exec_Master_Log_Pos: 114

mysql> show global variables like '%workers%';
+------------------------+-------+
+------------------------+-------+
| slave_parallel_workers | 0 |
+------------------------+-------+

mysql> SET GLOBAL slave_parallel_workers=4;
Query OK, 0 rows affected (0.00 sec)

mysql> show global variables like '%workers%';
+------------------------+-------+
+------------------------+-------+
| slave_parallel_workers | 4 |
+------------------------+-------+

http://dev.mysql.com/doc/refman/5.6/en/replication-options-slave.html#sysvar_slave_parallel_workers


Binary Log Group Commit

Binary
Log  Increases replication throughput by
increasing performance of the
T1 T2
Session
master
T3 T4  Commits multiple transactions as a
group to Binlog on disk
 Finer grained locking; reducing lock
wait times
Group
commit 155% Performance Gain
Master
Database


Optimized Row Base Replication

Primary Key Changed Columns
 New option: binlog-row-image=minimal
 Increases throughput for master and slave
- Reduces Binlog size, memory & network bandwidth
 Only replicates elements of the Row image that have changed


MySQL 5.6: Crash safe Slaves, BinLog
Problem: Slave thread commits and then updates
slave data.
what if it crashes in-between?
●
recovery issues - where to restart replication from?
●
file corruption vulnerability
●
administration hassle

Solution: store data in (transactional) tables:
●
slave's data is updated transactionally
●
engine agnostic
●
crash-safeness (requires transactional tables) frees
the DBA to manually recover transactions after a
crash

http://farm6.static.flickr.com/5024/5554971120_df447dd31c.jpg



Atomic Data • Automatic recovery of a slave and
CRASH! Binlog after a failure
Position Info Time • Binlog and table data are transactionally
Before:
consistent
– Transaction Data: in tables
– Replication Info: in files
• Resumes replication without Dev/Op
intervention
• Automatically rolling back replication to
Atomic Data
last committed event
• Eliminates risk of data loss or
Position Info
Time corruption
MySQL 5.6
– Transaction Data: in tables
– Replication Info: in tables


●
System tables:
slave_master_info (mysql.slave_master_info)
--master-info-repository=TABLE
slave_relay_log_info (mysql.slave_relay_log_info)
--relay-log-info-repository=TABLE

mysql_slave> stop slave;
mysql_slave> SET GLOBAL master_info_repository = 'TABLE';
mysql_slave> SET GLOBAL relay_log_info_repository = 'TABLE';
mysql_slave> start slave;
Make sure you add to my.cnf
– master-info-repository =TABLE
●
relay-log-info-repository =TABLE
●
Transactional tables enables transactional slave positions
●
Automatic conversion between files and tables on startup
●
Long time awaited feature

http://dev.mysql.com/doc/refman/5.6/en/replication-options-binary-log.html


mysql_slave> select * from slave_master_info G
*************************** 1. row ***************************
Master_id: 2
Number_of_lines: 22
Master_log_name: yoda-bin.000003
Master_log_pos: 323
Host: yoda
User_name: replication
User_password: slavepass
Port: 3306
Connect_retry: 10
Enabled_ssl: 0
Ssl_ca:
Ssl_capath:
Ssl_cert:
Ssl_cipher:
Ssl_key:
Ssl_verify_server_cert: 0
Heartbeat: 1800
Bind:
Ignored_server_ids: 0
Uuid: 75d407df-2be4-11e1-9668-b4be9bce39b0
Retry_count: 86400
Ssl_crl:
Ssl_crlpath:


MySQL 5.6: Replication Event Checksums

 Ensures replicated data is correct,
consistent and accessible
Master Slave
 Detects corrupt replication events
before they’re applied
– Returns an error
 Protects entire replication path
– Memory
– Disk
– Network
– Bugs



Problem: Make replication slave to lag a specified amount of time behind
the master to:
●
To protect against user mistakes on the master.
●
To test how the system behaves when there is a lag.
●
To inspect what the database looked like long ago, without having
to reload a backup.

Solution: The slave waits until a given number of seconds elapses before
applying the changes:
●
Delays configured per slave: flexible deployment;
●
Are implemented in the SQL thread layer.
●
Rolling Database Backups with Relayed Replication


User interface:
●
CHANGE MASTER TO MASTER_DELAY = <NUM_SECONDS>;
●
mysql> stop slave;
●
mysql> CHANGE MASTER TO MASTER_DELAY=86400; start slave;

●
SHOW SLAVE STATUS:
●
SQL_Delay: 86400
●
SQL_Remaining_Delay: 86395
●
Slave_SQL_Running_State: Waiting until MASTER_DELAY seconds after master
executed event
●
RESET SLAVE clears the configured delay;

Rolling forward delayed slaves until bad event:
START SLAVE [SQL_THREAD] UNTIL MASTER_LOG_FILE = 'log_name',
MASTER_LOG_POS = log_pos

http://dev.mysql.com/doc/refman/5.6/en/replication-delayed.html
http://dev.mysql.com/doc/refman/5.6/en/start-slave.html



slave2> CHANGE MASTER TO
-> MASTER_HOST = 'localhost',
-> MASTER_PORT = 3306,
-> MASTER_USER = 'repl_user',
-> MASTER_PASSWORD = 'pw',
-> MASTER_DELAY = 86400;

slave2> START SLAVE;


mysql_luke> STOP SLAVE;

mysql_luke>SHOW RELAYLOG EVENTS FROM 2337G
*************************** 4. row ***************************
Log_name: luke-relay-bin.000005
Pos: 2674
Event_type: Query
Server_id: 1
End_log_pos: 2623
Info: drop database tatooine

mysql_luke> START SLAVE UNTIL
-> MASTER_LOG_FILE='luke-relay-bin.000005',
-> MASTER_LOG_POS=2674;

http://www.minifigures.co.uk/lego/star-wars/darth-vader-minifig/


MySQL Enterprise Monitor

All the slaves paying attention?

●
Do you know when a slave is behind?

●
What type of topology do you have?

●
Alerted on Errors?

●
Slow Queries?

http://www.stormgrounds.com/fullwallpaper/320/Storm-Trooper-Legos



●
Global view of MySQL environment
●
Automated, rules-based monitoring and
alerts (SMTP, SNMP enabled)
●
Query capture, monitoring, analysis
and tuning, correlated with Monitor
graphs
●
Visual monitoring of “hot” applications
and servers
●
Real-time Replication Monitor with
auto-discovery of master-slave
topologies
●
Integrated with MySQL Support


MySQL Replication Monitoring


MySQL User Groups
Top Active MySQL User Groups
North America Japan and APAC South America
●
Atlanta, GA ●
Montreal, CAN ●
Taiwan ●
Brazil
●
Austin, TX ●
Minneapolis, MN ●
Indonesia ●
Indonesia
●
Boise, ID ●
New York, NY
●
Boston, MA ●
New Jersey, NJ Europe Oceania
●
Cleveland, OH ●
Philadelphia, PA ●
Helsinki ●
Sydney Australia
●
Denver, CO ●
Raleigh, NC ●
London
●
Dallas, TX ●
San Francisco, CA ●
France Middle East
●
Huntsville, AL ●
Seattle, WA ●
Dubai
●
Los Angeles, CA ●
Toronto, CAN Middle East ●
Turkey
●
Dubai
●
Turkey
Complete list of MySQL user groups
https://wikis.oracle.com/display/mysql/List+of+MySQL+User+
Groups

MySQL Community Managers
Dave Stokes (david.stokes@oracle.com)
Keith Larson (keith.larson@oracle.com)


MySQL Available Now
Development Releases:
●
http://dev.mysql.com/downloads/mysql/#downloads

Evaluate the new features
●
Questions & Feedback: forums.mysql.com/list.php?26
●
Bugs: bugs.mysql.com/

Replication in MySQL 5.5
●
mysql.com/why-mysql/white-papers/mysql-wp-replication.php

Replication 5.6 documentation
●
dev.mysql.com/doc/refman/5.6/en/replication.html

Planet
●
Planet.mysql.com

MySQL High Availability
●
http://shop.oreilly.com/product/9780596807290.do

Learn more about HA Solutions for MySQL
●
mysql.com/why-mysql/white-papers/mysql_wp_ha_strategy_guide.php
http://www.stormgrounds.com/wallpaper/Miscellaneous/Stormtrooper-under-Keyboard


Virtual Developer Day March 12

Virtual Developer Day: MySQL is a one-stop shop for you to
learn all the essential MySQL skills. With a combination of
presentations and hands-on lab experience, you’ll have the
opportunity to practice in your own environment and sharpen
your skills to:
• Develop your new applications cost-effectively using
MySQL
• Improve performance of your existing MySQL databases
• Manage your MySQL environment more efficiently

https://oracle.6connex.com/portal/mysql/login/?
langR=en_US&mcc=launch

David.Stokes@oracle.com
@stoker

http://www.flickr.com/photos/kalexanderson/7964300088/sizes/l/in/photostream/


<Insert Picture Here>

Thanks for attending!
keith.larson@oracle.com
sqlhjalp.blogspot.com
sqlhjalp.com/pdf/

Keith Larson Replication

More Related Content

What's hot (20)

Similar to Keith Larson Replication (20)

More from Dave Stokes (20)

Recently uploaded (20)

Keith Larson Replication