MHA (MySQL High Availability): Getting started & moving past quirks
21 likes6,632 views
MHA (Master High Availability Manager) is a tool for automating MySQL master failover and slave promotion with minimal downtime. It consists of Perl scripts that monitor a replication topology and automatically switch a slave to a master if the current master fails. When a failure is detected, MHA identifies the most up-to-date slave and applies any missing binary logs to make it the new master. It then changes the slaves to replicate from the new master. MHA requires installing node packages on all servers and a manager package to coordinate monitoring and failover.
![app1.cnf
[server default]
manager_workdir=/var/log/masterha/app1
manager_log=/var/log/masterha/app1/manager.log
[server1]
no need to specify master as
hostname=host1
MHA auto-detects this
[server2]
hostname=host2
sets priority, but doesn’t necessarily mean it gets promoted
candidate_master=1
as a default (say its too far behind replication).
But maybe this is a more powerful box, or has a better setup
[server3]
hostname=host3
[server4]
will never be the master. RAID0 instead of RAID1+0?
hostname=host4
Slave is in another data centre?
no_master=1](https://image.slidesharecdn.com/mhagettingstartedmovingpastquirks-130122021839-phpapp01/85/MHA-MySQL-High-Availability-Getting-started-moving-past-quirks-21-320.jpg)
![masterha_default.cnf
[server default]
check master activity from
manager->remote_hostN->
user=root
master (multiple hosts to
password=rootpass
ensure its not a network issue)
ssh_user=root
master_binlog_dir= /var/lib/mysql,/var/log/mysql
remote_workdir=/data/log/masterha
ping_interval=3
# secondary_check_script=masterha_secondary_check -s remote_host1 -s
remote_host2
# master_ip_failover_script= /script/masterha/master_ip_failover
# shutdown_script= /script/masterha/power_manager
# report_script= /script/masterha/send_report
# master_ip_online_change_script= /script/masterha/master_ip_online_change](https://image.slidesharecdn.com/mhagettingstartedmovingpastquirks-130122021839-phpapp01/85/MHA-MySQL-High-Availability-Getting-started-moving-past-quirks-22-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to MHA (MySQL High Availability): Getting started & moving past quirks (20)
Ad
More from Colin Charles (20)
Recently uploaded (20)
MHA (MySQL High Availability): Getting started & moving past quirks
- 1. MHA: Getting Started & Moving Past Quirks Colin Charles, Monty Program Ab [email protected] | [email protected] http://montyprogram.com/ | http://mariadb.org/ http://bytebot.net/blog/ | @bytebot on Twitter Percona Live London 2012, London, United Kingdom 4 December 2012
- 2. whoami • Chief Evangelist, MariaDB at Monty Program Ab • Formerly MySQL AB/Sun Microsystems • Past lives included Fedora Project (FESCO), OpenOffice.org
- 3. Aims • Why MHA? • What does MHA do? • How does MHA do it? • Running MHA,VIP failover, integration, etc • Who uses MHA?
- 4. My experience with MHA • Started November 2011 (MHA 0.52, 0.53) • NRE work to make it run on Solaris 10 environment... with no Internet access :-) • Continued deployment advice+work for data centre use • Much thanks to SkySQL for the experience
- 5. Why this talk • High Performance MySQL, 3rd Edition • Published: March 16 2012
- 6. Where did MHA come from? • 2011 MySQL Community Award Corporate Contributor: DeNA • MHA came shortly after the award • Written by Yoshinori Matsunobu, Oracle ACE Director
- 7. What is MHA? • MHA for MySQL: Master High Availability Manager tools for MySQL • Goal: automating master failover & slave promotion with minimal downtime • Set of Perl scripts • http://code.google.com/p/mysql-master-ha/
- 8. Why MHA? 1. Automating monitoring of your replication topology for master failover 2. Scheduled online master switching to a different host for online maintenance • switch back after OPTIMIZE/ALTER table, software or hardware upgrade 3. Interactive/non-interactive master failover (just for failover, with detection of master failure + VIP takeover to Pacemaker)
- 9. Why is master failover hard? • When • slave2 is master fails, latest, no more slave1+3 writes till have missing failover events, complete MHA does: • MySQL • copy replication id=10 is from asynchrono master if us (MHA possible works with async + • apply all semi-sync missing replication) events
- 10. Semi-sync replication • if master crashes (or is shutdown :P), MHA cannot save binlogs, latest data is lost • minimizes risk of binlog event existing only on crashed master • guarantees at least one slave receives binlog events at commit • http://code.google.com/p/mysql-master-ha/wiki/ UseCases#Using_together_with_Semi- Synchronous_Replication
- 11. MHA: Typical scenario • Monitor replication topology • If failure detected on master, immediately switch a candidate master or the most current slave to become new master • CHANGE MASTER for all slaves to new master • Print (stderr)/email report, stop monitoring
- 12. What does MHA do?
- 13. Typical timeline • Usually no more than 10-30 seconds • 0-10s: Master failover detected in around 10 seconds • (optional) 10-20s: 10 seconds to power off master • 10-20s: apply differential relay logs to new master • Practice: 4s @ DeNA, usually less than 10s
- 14. How does MHA work? • Save binlog events from crashed master • Identify latest slave • Apply differential relay log to other slaves • Apply saved binlog events from master • Promote a slave to new master • Make other slaves replicate from new master
- 15. Getting Started • MHA • MHA does requires no not build changes to replication your environmen application ts for you - that’s DIY • You are of course to write to a virtual IP (VIP) for your master
- 16. MHA Node • Download mha4mysql-node & install this on all machines: master, slaves, monitor • Packages (DEB, RPM) available • Manually, make sure you have DBD::mysql & ensure it knows the path of your MySQL
- 17. What’s in MHA node • save_binary_logs - save & copy master’s binlogs • apply_diff_relay_logs - find differential relay log events & apply missing events • purge_relay_logs - purge relay log files • filter_mysqlbinlog - OBSOLETE + not used by MHA (trims ROLLBACK statements & equivalent binlog events added by mysqlbinlog)
- 18. MHA Manager server • Monitor server doesn’t have to be powerful at all, just remain up • This is a single-point-of-failure so monitor the manager server where MHA Manager gets installed • If MHA Manager isn’t running, your app still runs, but automated failover is now disabled
- 19. MHA Manager • You must install mha4mysql-node then mha4mysql-manager • Manager server has many Perl dependencies: DBD::mysql, Config::Tiny, Log::Dispatch, Parallel::ForkManager, Time::HiRes • Package management fixes dependencies, else use CPAN
- 20. Configuring MHA • Application configuration file: see samples/conf/app1.cnf • Place this in /etc/MHA/app1.cnf • Global configuration file: see /etc/MHA/ masterha_default.cnf (see samples/conf/ masterha_default.cnf)
- 21. app1.cnf [server default] manager_workdir=/var/log/masterha/app1 manager_log=/var/log/masterha/app1/manager.log [server1] no need to specify master as hostname=host1 MHA auto-detects this [server2] hostname=host2 sets priority, but doesn’t necessarily mean it gets promoted candidate_master=1 as a default (say its too far behind replication). But maybe this is a more powerful box, or has a better setup [server3] hostname=host3 [server4] will never be the master. RAID0 instead of RAID1+0? hostname=host4 Slave is in another data centre? no_master=1
- 22. masterha_default.cnf [server default] check master activity from manager->remote_hostN-> user=root master (multiple hosts to password=rootpass ensure its not a network issue) ssh_user=root master_binlog_dir= /var/lib/mysql,/var/log/mysql remote_workdir=/data/log/masterha ping_interval=3 # secondary_check_script=masterha_secondary_check -s remote_host1 -s remote_host2 # master_ip_failover_script= /script/masterha/master_ip_failover # shutdown_script= /script/masterha/power_manager # report_script= /script/masterha/send_report # master_ip_online_change_script= /script/masterha/master_ip_online_change
- 23. MHA uses SSH • MHA uses SSH actively; passphraseless login • In theory, only require Manager SSH to all nodes • however, remember masterha_secondary_check • masterha_check_ssh --conf=/ etc/MHA/app1.cnf
- 24. Check replication • masterha_check_repl --conf=/etc/MHA/app1.cnf • If you don’t see MySQL Replication Health is OK, MHA will fail • Common errors? Master binlog in different position, read privileges on binary/relay log not granted, using multi-master replication w/o read- only=1 set (only 1 writable master allowed)
- 25. MHA Manager • masterha_manager --conf=/etc/MHA/app1.cnf • Logs are printed to stderr by default, set manager_log • Recommended running with nohup, or even daemontools • http://code.google.com/p/mysql-master- ha/wiki/Runnning_Background
- 26. So, the MHA playbook • Install MHA node, MHA manager • masterha_check_ssh --conf=/etc/app1.cnf • masterha_check_repl --conf=/etc/app1.cnf • masterha_manager --conf=/etc/app1.cnf • That’s it!
- 27. master_ip_failover_script • Pacemaker can monitor & takeover VIP if required or you can have a catalog database • Shared VIP is easy to implement with minimal changes to master_ip_failover itself (however, use shutdown_script to power off machine)
- 28. master_ip_online_change • similar to master_ip_failover script, but used for online maintenance •masterha_master_switch -- master_state=alive • MHA executes FLUSH TABLES WITH READ LOCK after the writing freeze
- 29. Test the failover! •masterha_check_status -- conf=/etc/MHA/app1.cnf • Kill MySQL (kill -9, shutdown server, kernel panic) • MHA should go thru failover (stderr) • parse the log as well • Upon completion, it stops running
- 30. masterha_master_switch • manual failover • --master_state=dead • scheduled online master switchover • great for upgrades to server, etc. • masterha_master_switch -- master_state=alive --conf=/etc/MHA/ app1.cnf --new_master_host=host2
- 31. Handling VIPs master_ip_online_change my $vip = ‘192.168.0.1/24”; master_ip_failover my $interface = “0”; my $ssh_start_vip = “sudo /sbin/ifconfig eth0:$key $vip”; my $ssh_stop_vip = “sudo /sbin/ifconfig eth0:$key down”; ... sub start_vip() { `ssh $ssh_user@$new_master_host ” $ssh_start_vip ”`; } sub stop_vip() { `ssh $ssh_user@$orig_master_host ” $ssh_stop_vip ”`; }
- 32. Integration with other HA solution • Pacemaker • on RHEL6, you need some HA add-on, just use the CentOS • /etc/ha.d/haresources to configure VIP • A mostly 80% guide on how to get it going (using corosync+pacemaker): • http://www.mysqlfanboy.com/2012/02/the- full-monty-version-2-3/
- 33. Solaris 10 • MHA tested to work on Linux and Solaris 10 & greater • Use a .pkg MySQL • Solaris 10 needs a compiler (SolarisStudio - register w/ Oracle to download) • CPAN: manual builds of dependencies • Solaris doesn’t have md5sum (call md5) • Solaris ssh isn’t OpenSSH, missing some features like ConnectionTimeout
- 34. What about replication delay? • by default, MHA checks to see if slave is behind master. By more than 100MB, it is never a candidate slave • If you have candidate_master=1 set, consider setting check_repl_delay=0 • You can integrate it with pt-heartbeat from Percona Toolkit • http://www.percona.com/doc/percona-toolkit/2.1/ pt-heartbeat.html
- 35. MHA deployment tips • You really should install this as • You can have a standby master, root make sure its read-only • SSH needs to work across all • By default, master1->master2- hosts >slave3 doesn’t work • If you don’t want plaintext • MHA manages master1- passwords in config files, use >master2 w/o issue init_conf_load_script • use multi_tier_slave=1 • Each monitor can monitor option multiple MHA pairs (hence app1, app2, etc.) • Make sure replication user exists on candidate master too!
- 36. Alternative solutions • Heartbeat + DRBD • cost $$$ -> passive master • innodb_flush_log_at_trx_commit=1, sync_binlog=1 • use MariaDB/PS: https:// kb.askmonty.org/en/group-commit-for- the-binary-log/
- 37. Alternative solutions II • MySQL NDB Cluster • its not InnoDB... • Percona Replication Manager (PRM) • Tungsten Replicator • Eventually (maybe), mysqlfailover from mysql-utilities using GTIDs in 5.6+
- 38. Where is MHA used • DeNA • Premaccess (Swiss HA hosting company) • Ireland’s national TV & radio service • Jetair Belgium (MHA + MariaDB!) • Samsung • SK Group • DAPA • Facebook
- 39. MHA 0.54 Just released! • MHA is being worked on & developed while Matsunobu-san is at Facebook • MHA 0.54 is released December 1 2012 • http://code.google.com/p/mysql-master-ha/ wiki/ReleaseNotes
- 40. Roadmap • Improve examples in package (0.55 is already in works) • Make more integrations: pt-heartbeat, commit Solaris code upstream, easier packages with Pacemaker/corosync • More automated master recovery if possible • Front-end that isn’t console
- 41. Support • SkySQL: www.skysql.com • Training from AccelerationDB • http://www.accelerationdb.com/mysql- mha-training.htm • Developer support from Monty Program Ab: http://montyprogram.com/developer- support/
- 42. Automated tools • 4-host Vagrant setup for MySQL MHA: https://github.com/hholzgra/vagrant-mysql- mha • Palomino Cluster Tool: https://github.com/ time-palominodb/PalominoClusterTool • Ansible playbooks for MHA
- 43. Video • Yoshinori Matsunobu talking about High Availability & MHA at Oracle MySQL day: http://www.youtube.com/watch? v=CNCALAw3VpU • Alex Alexander (AccelerationDB) talks about MHA, with an example of failover, and how it compares to Tungsten: http:// www.youtube.com/watch?v=M9vVZ7jWTgw
- 44. References • Design document: http://www.slideshare.net/ matsunobu/automated-master-failover • Configuration parameters: http://code.google.com/ p/mysql-master-ha/wiki/Parameters • JetAir MHA use case: http://www.percona.com/ live/mysql-conference-2012/sessions/case-study- jetair-dramatically-increasing-uptime-mha • MySQL binary log: http://dev.mysql.com/doc/ internals/en/binary-log.html
- 45. Advert • Donate to the MariaDB project & help keep the ecosystem alive • https://kb.askmonty.org/en/ donations/ • Buy some merchandise: • http://www.cafepress.com/ mariadb
- 46. Q&A [email protected] | [email protected] http://montyprogram.com/ | http://mariadb.org/ twitter: @bytebot | url: http://bytebot.net/blog/