CoreOS automated MySQL Cluster Failover using Galera Cluster
5 likes5,356 views
CoreOS Fleet and Etcd provide a simple and eloquent framework for application clusters to both auto-configure and recover from node failure. Galera Cluster is a multi-master, open solution for clustering MySQL. Mix the two, sprinkle in a bit of “glue” and you have a Docker based MySQL cluster that will react automatically to container failure. This presentation will cover the nuts and bolts of automating a Galera Cluster, built from Docker Images and deployed in a distributed fashion using etcd, confd, and fleet for both initial and failure recovery configuration.
![[Devconf.cz][2017] Understanding OpenShift Security Context Constraints](https://cdn.slidesharecdn.com/ss_thumbnails/devconf-170124111354-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (10)
Ad
Similar to CoreOS automated MySQL Cluster Failover using Galera Cluster (20)
Ad
Recently uploaded (20)
CoreOS automated MySQL Cluster Failover using Galera Cluster
- 1. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. CoreOSautomatedMySQL ClusterFailoverusingGaleraCluster Hewlett-Packard (Enterprise) Advanced Technology Group – Cloud Yazz Atlas Patrick Galbraith Spring 2015
- 2. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Aboutthespeaker ● Yazz Atlas ● HP Advanced Technology Group ● HP Cloud Services PaaS/DBaaS/ LBaaS Environment ● Has worked at Windermere Real Estate, Keas, Slashdot, Cobalt Group, University of Iowa, Antioch College
- 3. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Aboutthespeaker ● Patrick Galbraith ● HP Advanced Technology Group ● Has worked at Blue Gecko, MySQL AB, Classmates, Slashdot, Cobalt Group, US Navy, K-mart ● MySQL projects: memcached UDFs, DBD::mysql, federated storage engine ● Family ● Outdoors
- 4. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4 Purpose of this talk – why are you here? Basic Pre-req • Overview of the Demo Setup Docker • Containers vs. Virtulization • Simple Docker usage • Clustered Docker CoreOS • Container optimized, stripped down Linux distribution • Overview of core components of CoreOS – fleet, etcd, systemd Galera • Synchronous replication – excellent solution to clustering MySQL • HA Proxy for failover Discussion • What is next • What about Kubernetes
- 5. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5 Laptop Host - VirtualBox - OpenStack - Vmware - Bare metal - et al. Master-01 Basic Pre-reqs Setup CoreOS + etcd + confd Some of the times you will need • CoreOS • Go – https://golang.org/doc/install • Etcdctl – go get github.com/coreos/etcd/etcdctl • Fleetctl – go get github.com/coreos/fleet/fleetctl Core-01 Core-02 Core-03
- 6. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6 What are containers? • Operating-system-level virtualization • Encapsulated, hermetically sealed applications • Relatively isolated • Small footprint • Fast to launch! • Portable. And did I mention, portable?! • Use of host OS and Kernel • Execution consists of time to startup application in question • LXC, Docker, Solaris Zones, BSD Jails, Parallels Virtuozzo, OpenVZ, … Photo credit: expo_2020 - http://goo.gl/OeSQhi License CC v2.0
- 7. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 Containers vs. VMs
- 8. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8 Docker What is Docker? • Set of tools for managing containers • Command line tool that doubles as a daemon • Uses Linux Kernel Features: – Kernel namespaces – the core ingredient to containers working: PID, IPC ,uts (what will be seen by a group of processes), mount, network and user – Cgroups (control groups) -- limit, account and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups • Originally used lxc, now defaults to Libcontainer but meant for any containerization mechanism • Much more light weight than VMs • Encapsulated application containers in a relatively isolated but lightweight operating environment • Written in Go
- 9. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9 Clustered Docker • CoreOS (https://coreos.com/) • Kubernetes (http://kubernetes.io) • Mesos + Marathon (http://mesos.apache.org/) Apache project, Zookeeper, etc • Project Atomic (http://www.projectatomic.io/) RH/Fedora/Centos designed for running Docker • Docker Openstack (https://wiki.openstack.org/wiki/Docker) Hypervisor Driver for Openstack Compute • Swarm/Compose/Machine • RancherOS (http://rancher.com/rancher-os) Minimalist Linux, Docker daemon runs as PID 1 first process the kernel starts known as “System Docker,” • Flocker (https://clusterhq.com) • Spotify Helios (https://github.com/spotify/helios) Zookeeper • Flynn (https://flynn.io/) • Deis (http://deis.io) • Maestro (https://github.com/toscanini/maestro) • Shipyard (http://shipyard-project.com) • … others to come!
- 10. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10 What is CoreOS? CoreOS • Minimalist Linux • Optimized for containers • Easy to run containers • Service discovery, container management, • Docker -- Container runtime and management, though Rocket long-term • Etcd – distributed global key value store for config data on each node • Fleet – Rudimentary Scheduler interacts with systemd and etcd • Systemd – Newer Linuxi – system and service manager for Linux • Flannel – Networking across nodes
- 11. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11 Connecting it all together CoreOS • Etcd • Fleetd • Flanneld Photo credit: Stéfan – http://goo.gl/ye8Icd License CC v2.0
- 12. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12 Galera secret sauce • Galera (Percona XtraDB Cluster) container • FLEETCTL_ENDPOINT set to ETCD host:port • etcd included • Entrypoint script • Generates SQL file to create SST (single state transfer) user and clustercheck user • Builds cluster.cnf • Wsrep_node_address set to its own IP • Wsrep_cluster_address set using entries from etcd • Server-id random number • clustercheck on each galera node container (port 8000) to determine health and which nodes available using clustercheck user to connect • Ports 3306, 4444, 4567, 4568, 8000 exposed provides access to Galera cluster node containers for external and HAProxy container
- 13. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13 Galera secret sauce – haproxy sprinkling • HAProxy container • FLEETCTL_ENDPOINT set to ETCD host:port • Confd included • Confd builds haproxy.cfg from template using etcd entries for each Galera node • Confd loop entered to start container • HAProxy uses clustercheck on each node (port 8000) to determine health and which nodes available • Port 33006 exposed provides access to Galera cluster node containers
- 14. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14 Laptop Virtualbox Master-01 Core-01 Core-02 Core-03 etcd flanneld Galera-1 Galera-2 Galera-3 haproxy Sidekick-1 Sidekick-2 Sidekick-2 Flannel network Galerareplication HAProxy etcdcommunication Connecting it all together. $> vagrant up
- 15. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15 Galera on CoreOS
- 16. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16 Disaster Strikes! Don’t put all your containers on one ship (CoreOS host). • HA Proxy is the middleman connecting to your Galera Cluster – Minimal impact to connecting agents • Adding a new CoreOS host and redeploy the Galera container Photo credit Paul Townsend – http://goo.gl/ImL8hX License CC v2.0
- 17. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17 Demo time… DEMO • Vagrant - 1.7.2 • Virtualbox - 4.3.26r98988 – coreos-alpha (virtualbox, 660.0.0) • Github Repos – https://github.com/EntropyWorks/coreos-vagrant – https://github.com/EntropyWorks/fleet-units-galera-cluster – https://github.com/EntropyWorks/docker_haproxy – https://github.com/CaptTofu/percona_xtradb_cluster_docker (coreos branch) • Asciinema recordings – https://asciinema.org/a/19559
- 18. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. ThankYou Hewlett-Packard, Advanced Technology Group Yazz Atlas [email protected] Patrick Galbaith [email protected]
Editor's Notes
- #11: Using coreos as a container-optimized VM