High Availability in 37 Easy Steps
Download as ODP, PDF16 likes6,884 views
High Availability can be a curiously nebulous term, and most people probably don't care about it until they can't access their online banking service, or their plane crashes. This presentation examines some of the considerations necessary when building highly available computer systems, then focuses on the HA infrastructure software currently available from the Corosync/OpenAIS, Linux-HA and Pacemaker projects. Originally presented at Linux Users Victoria in April 2010 (http://luv.asn.au/2010/04/06)
![High Availability in 37 Easy Steps Tim Serong Senior Clustering Engineer [email_address]](https://image.slidesharecdn.com/highavailabilityin37easysteps-100407061928-phpapp02/85/High-Availability-in-37-Easy-Steps-1-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
![Drp International Brochure Version 5.5[1]](https://cdn.slidesharecdn.com/ss_thumbnails/DRPInternationalBrochureversion551-123319755553-phpapp01-thumbnail.jpg?width=560&fit=bounds)
Ad
Similar to High Availability in 37 Easy Steps (20)
Ad
Recently uploaded (20)
High Availability in 37 Easy Steps
- 1. High Availability in 37 Easy Steps Tim Serong Senior Clustering Engineer [email_address]
- 2. Agenda What is High Availability?
- 5. What is High Availability?
- 6. What is High Availability? “ High availability is a system design protocol and associated implementation that ensures a certain degree of operational continuity during a given measurement period.” http://en.wikipedia.org/wiki/High_Availability
- 7. So: Increase MTTF (better hardware)
- 8. Decrease MTTR (redundant hardware + software) What is High Availability? Availability = MTTF MTTF + MTTR
- 9. What is High Availability? (hopefully your hardware is better than this)
- 10. What is High Availability? I lied about the presentation title
- 11. What is High Availability? I lied about the presentation title
- 12. High Availability in 37 Easy Steps
- 13. What is High Availability? I lied about the presentation title
- 14. High Availability in 37 Easy Steps
- 15. High Availability is a Process, not a Product
- 16. What is High Availability? (hopefully you hired this sysadmin) http://xkcd.com/705/
- 18. System Design Considerations What, exactly, do you need?
- 19. How good is your system already?
- 20. Within what limits can you operate?
- 21. Please, for the love of Eris, keep it simple.
- 22. System Design Considerations Dual F/C Ethernet RAID File Server Client Network
- 23. System Design Considerations Dual F/C Ethernet Reasonably Highly Available, Most of the Time RAID File Server Client Network
- 24. System Design Considerations Good: Redundant power to server
- 26. RAID Bad: Server can still fail
- 27. Software can still fail
- 28. System Design Considerations Dual F/C Ethernet Dual F/C Ethernet Private Network “ File Server” RAID Node 1 Client Network Node 2
- 29. System Design Considerations Dual F/C Ethernet Dual F/C Ethernet Private Network “ File Server” Node 2 takes over when Node 1 fails RAID Node 1 Client Network Node 2
- 30. System Design Considerations Redundancy adds complexity
- 31. Who's the boss if the two nodes get confused?
- 32. STONITH to the rescue
- 34. System Design Considerations Two-node clusters can be problematic.
- 35. Set STONITH action to power off (not reset).
- 36. Get a third node.
- 37. Test, test , test!
- 39. HA Clustering Software In the beginning was Heartbeat v1
- 40. Easy to configure... # cat /etc/haresources
- 41. node1 IPaddr::192.168.1.50 Filesystem::/dev/sda1::/data1::ext3 ...because it couldn't do anything. – http://theclusterguy.clusterlabs.org/post/178680309/configuring-heartbeat-v1-was-so-simple
- 42. HA Clustering Software Then came Heartbeat v2, supporting: More than 2 nodes
- 45. HA Clustering Software Trickier to configure: # cibadmin -Q <?xml version="1.0"?> <cib ...> <configuration> ... <resources> <group id="ip-with-fs"> <primitive class="ocf" id="IP" provider="heartbeat" type="IPaddr"> <instance_attributes id="IP-instance_attributes"> <nvpair id="IP-instance_attributes-ip" name="ip" value="192.168.1.50"/> </instance_attributes> <operations> <op id="IP-monitor-5min" interval="5min" name="monitor"/> </operations> </primitive> <primitive class="ocf" id="FS" provider="heartbeat" type="Filesystem"> <instance_attributes id="FS-instance_attributes"> <nvpair id="FS-instance_attributes-device" name="device" value="/dev/sda1"/> <nvpair id="FS-instance_attributes-directory" name="directory" value="/data1"/> <nvpair id="FS-instance_attributes-fstype" name="fstype" value="ext3"/> </instance_attributes> <operations> <op id="FS-monitor-60s" interval="60s" name="monitor"/> </operations> </primitive> </group> </resources> <constraints> <rsc_location id="prefer-node1" node="node1" rsc="ip-with-fs" score="100"/> </constraints> ... </configuration> </cib>
- 46. HA Clustering Software Sometimes, it hurts your eyes:
- 47. HA Clustering Software Heartbeat then split into two projects: Heartbeat 2.1.4 (membership & messaging)
- 49. (also glue, agents) Pacemaker added support for OpenAIS as an alternative to Heartbeat Necessary for DLM, thus supporting CLVM, GFS2, OCFS2
- 50. HA Clustering Software http://clusterlabs.org/wiki/Architecture
- 51. HA Clustering Software Most recently, OpenAIS split into two projects: Corosync (cluster infrastructure)
- 52. OpenAIS (SA Forum APIs, i.e. magic for DLM, OCFS2, etc.) So now we have: Pacemaker 1.x on Heartbeat 3.0, or,
- 53. Pacemaker 1.x on Corosync 1.x (+ OpenAIS 1.x)
- 54. HA Clustering Software Linux Kernel (SUSE only) (diagram courtesy of Lars Marowsky-Brée) ext3, XFS OCFS2 cLVM2 Local Disks SAN FC(oE), iSCSI DRBD Multipath IO DLM SCTP TCP UDP multicast UDP multicast Ethernet Infiniband Bonding SAP MySQL libvirt Xen Apache iSCSI Filesystems IP address DRBD clvmd Ocfs2_controld dlm_controld YaST2 c DRBD c OpenAIS MPIO LVS Resource Agents LSB init STONITH LRM ... DRAC iLO SBD Fencing Web GUI Python GUI CRM Shell CIB Policy Engine Pacemaker OpenAIS
- 55. HA Clustering Software But, after all that, it's easy to configure again:
- 56. ...and vastly more flexible # crm configure show primitive IP ocf:heartbeat:IPaddr \ params ip="192.168.1.50" \ op monitor interval="5min" primitive FS ocf:heartbeat:Filesystem \ params device="/dev/sda1" directory="/data1" fstype="ext3" \ op monitor interval="60s" group ip-with-fs IP FS location prefer-node1 ip-with-fs 100: node1
- 57. HA Clustering Software Network Links Clients Storage (diagram courtesy of Lars Marowsky-Brée) Kernel Xen VM 1 LAMP Apache IP ext3 Kernel Kernel Corosync + openAIS Pacemaker DLM cLVM2+OCFS2 Xen VM 2
- 58. HA Clustering Software Before it breaks: # crm
- 59. crm(live)# cib new sandbox
- 60. INFO: sandbox shadow CIB created
- 61. crm(sandbox)# cib cibstatus load live
- 62. crm(sandbox)# cib cibstatus op monitor IP not_running
- 64. ptest[12971]: 2010/04/05_07:43:36 WARN: unpack_rsc_op: Processing failed op IP_monitor_300000 on hex-14: not running (7)
- 66. HA Clustering Software After it breaks: # hb_report -n “node1 node2” -f 12:00 /tmp/hb_report Compiles: Cluster-wide log files
- 67. Package state
- 68. DLM/OCFS2 state
- 70. CIB history
- 71. Parses core dump reports (needs debuginfo packages!)
- 72. ...into a single tarball for subsequent analysis
- 73. HA Clustering Software Active Community #linux-ha
- 74. #linux-cluster
- 75. Various mailing lists SUSE and Red Hat converging on cluster stacks
- 76. Heartbeat in maintenance mode