Nagios Conference 2011 - Jeff Sly - Case Study Nagios @ Nu Skin
Download as PPTX, PDF2 likes2,256 views
Jeff Sly, a Principal IT Architect at Nu Skin, discusses the company's history with monitoring systems, emphasizing the need for a centralized monitoring solution that accommodates various tools and platforms. The case study outlines a transition to Nagios as a 'Manager of Monitors' (MoM), enabling improved alert management and operational efficiency across six global data centers. The document includes insights on implementation methodology, tool requirements, and plans for future enhancements to their monitoring strategy.
![Tune the Database Modify contents of the /etc/my.cnf [mysqld] section.tmp_table_size=524288000max_heap_table_size=524288000table_cache=768set-variable=max_connections=100wait_timeout=7800query_cache_size = 12582912query_cache_limit=80000thread_cache_size = 4join_buffer_size = 128Khttp://web3us.comInfo on: MySQL Tuning, Nagios Tuning](https://image.slidesharecdn.com/jeffsly-casestudynagiosnuskin-111003123653-phpapp02/85/Nagios-Conference-2011-Jeff-Sly-Case-Study-Nagios-Nu-Skin-39-320.jpg)
Nagios Conference 2011 - Jeff Sly - Case Study Nagios @ Nu Skin
- 1. Jeff SlyPrincipal IT [email protected] Study Nagios @ Nu Skin
- 2. Who is in the Audience?How many of you are:Suppliers of Nagios or some value add-on for Nagios?Customers using Nagios?Just implementing Nagios or expanding implementation?Using NagiosXI?
- 3. Who is Nu Skin?
- 4. Our Technology FootprintEcommerce – Home grown Applications – Java, EJB, ABAP, .NetDatabases – Oracle, MySQL, MSSQLOS – HPUX, Redhat, Windows, VMWareERP – SAP Supply Chain, CRM, FIDatacenters – 6 locations in 6 countriesOffices – 50 Countries
- 5. Monitoring GoalsMonitoring presents operations with a completely integrated global view. Good monitoring is proactive; it helps teams prevent problems from becoming outages. Good monitoring helps minimize outage downtime, quickly identify root cause and contacts correct people.
- 7. Our Monitoring HistoryWe tried for 10 years…
- 8. Do it all in ‘One Tool Projects’One Monitoring Tool to rule them all:Mercury SiteScopeRemedy Help DeskHP OpenViewQuest FoglightHome grown (several)One monitoring personHe decided to quit!
- 9. Could never get everythingAll Failed – We always gave up! Why?Servers and agents that were proprietaryHuge foot print inefficient performanceSteep learning curveVery expensiveUpdates costly and very time consumingSystem Administrators like their own scripts, can see what they are doing
- 10. Resulting Monitoring IssuesTried to make Operations clearing house for all warnings and alerts from 10+ toolsOperations was overwhelmed Took 4 process steps and lots of software to notify of critical failuresMost Administrators setup own private monitoring to receive warningsMany false notificationsLate notifications
- 11. As Is (start of project)Our Business Customers were Unhappy
- 12. Old Monitoring Work FlowFour steps to notify system administrator
- 13. Step 1: Everything Emails OperationsEmail HelpDeskErrorNetworkHP NNMSystemScriptsNagiosDatabaseFoglightSiteScope 8BACSitescope 6
- 14. Step 2: Operations Opens EmailEmail HelpDeskErrorNetworkHP NNMSystemScriptsNagiosDatabaseFoglightSiteScope 8BACSitescope 6
- 15. Step 3: Operations Checks SourceEmail HelpDeskErrorNetworkHP NNMSystemScriptsNagiosDatabaseFoglightSiteScope 8BACSitescope 6
- 16. Step 4: Operations Calls adminEmail HelpDeskErrorNetworkHP NNMSystemScriptsNagiosDatabaseFoglightSiteScope 8BACSitescope 6
- 17. Inventory of Existing ChecksRegular Expression found on Web Page Monitoring HTTP Check - Up or Down Ping Host Up or Down PORT monitoring FTP checking SMTP checking SNMP monitoring - no trap catching yet RadiusDNS monitoring Disk Space monitoring CPU and Load Average monitoringMemory Monitoring
- 18. Inventory of Existing ChecksService monitoring Transaction monitoring - page load times – performance graph Website click through (Webinject not working) Log File monitor –parse for ErrorsJava HEAP, Thread, Threadlock monitoring Apache thread and worker count monitors Ecommerce shop monitorsEmail can send and receive SQL query ODBC (catalog ODBC had bugs)
- 20. Key IdeasMoMTool RequirementsShared OwnershipLowest LevelNagios Monitor Method
- 21. Idea 1: MoMOur first “break though” was the idea that even through we needed a centralized view for all monitoring that did not mean all monitoring had to be done by one monitoring tool. We had to pick a “Manager of the Monitors” (MoM) to bring together the best of breed monitoring.
- 22. MoM - according to Gartner
- 23. Idea 2: Tool RequirementsOpen – not proprietary and closedMainstream – wanted good native support and strong communityInterface – to 3rd Party MonitoringFlexible – adapt to many types of monitoringEfficient – minimal foot print on production servers, not chatty on networkNotification – granular controlReliable – good clean architectureUsability – GUI interface, reporting
- 24. Idea 3: Shared OwnershipCore teamOperation of Monitoring Environment: backups, upgrades, & custom plug-insMonitoring ExpertsTrainingMonitoring leads in Development & Admin teams:Set up own monitorsKeep own monitors currentAdjust monitorsIf something is not monitored not core teams fault
- 25. Operations Owned MonitoringEmail HelpDeskErrorNetworkHP NNMSystemScriptsNagiosDatabaseFoglightSiteScope 8BACSitescope 6
- 26. Team Leads Own MonitoringOperationsNetworkAsiaSystemScriptsEuropeSAPDatabaseWeb
- 27. How to Guides
- 29. How to Setup NRPE - HPUX
- 30. Idea 4: Lowest Level Handle alerts at the lowest possible level in the organizationOnly forward alerts if not handled at lower levels before they become critical
- 31. Handle events at lowest levelOperationsNetworkAsiaSystemScriptsEuropeSAPDatabaseWeb
- 32. Only forward unhandled alertsNetworkAsiaSystemScriptsEuropeSAPDatabaseWeb
- 33. Idea 5: Nagios Monitor MethodChoose the Nagios Monitoring MethodActive Check from Nagios Server (normal)Active Check performed by remote clientNRPE, NSClientPassive Check – Listen to 3rd party monitorsNSCA
- 34. Active Local CheckWebHTTP or PingNagiosDBMonitorUnixDBWin
- 35. Active Remote Check - UXWebNagiosCPU, RAM(NRPE)DBMonitorUnixDBWin
- 36. Active Remote Check - WinWebNagiosDBMonitorCPU, RAM(NSClient)UnixDBWin
- 37. Passive 3rd Party AlertWebNagiosDBMonitor3rd Party Alert NSCAUnixDBWin3rd Party Check DB
- 38. Bonus Idea - TuneTune the databaseAdd Ram Drive
- 39. Tune the Database Modify contents of the /etc/my.cnf [mysqld] section.tmp_table_size=524288000max_heap_table_size=524288000table_cache=768set-variable=max_connections=100wait_timeout=7800query_cache_size = 12582912query_cache_limit=80000thread_cache_size = 4join_buffer_size = 128Khttp://web3us.comInfo on: MySQL Tuning, Nagios Tuning
- 40. RAM DriveCreate a RAM disk for Nagios tempory filesI created a ramdisk by adding the following entry to the /etc/fstab file: none /mnt/ram tmpfs size=500M 0 0Mount the disk using the following commands # mkdir -p /mnt/ram; mount /mnt/ramVerify the disk was mounted and created # df -kModify the /usr/local/nagios/etc/nagios.cfg file with the following tuned parameterstemp_file=/mnt/ram/nagios.tmptemp_path=/mnt/ramstatus_file=/mnt/ram/status.datprecached_object_file=/mnt/ram/objects.precacheobject_cache_file=/mnt/ram/objects.cache
- 41. Implementation MethodologySite SurveyInventory existing monitorsProof of conceptBuild new environmentMigrate monitors from each platform to Nagios, one at a timeIntegrate OEM, and to send monitors to Nagios
- 42. Three Project PhasesDeliver something useful in each phaseBuild a level at a time
- 43. Phase ISet up a pilot of Nagios XI using Trial License. Set up Foglight monitoring of JVM (Java Virtual Machine). Purchase NagiosXI and Consulting SupportBring in a consultant for two weeks to help set up the architecture and help us work with the system. Documentation Web Site for Nagios learning's and “How to guides”Define a set of standards and guidelines to follow to help aid an effective monitoring process.Backups on Running on Production Nagios ServerSet up services which aren't being caught right now and move a few of the important services over to the new Nagios XI monitoring system. Test Nagios plugins and server performance
- 44. Phase IIMigrate off of Sitescope 6 and shutdownMigrate off of Sitescope 8 and shutdownDecommission FoglightClean up the old monitoring serverMigrate the network team from old Nagios to core NagiosXI systemSet up standby NagiosXI system, cron to replicate weeklyResearch missing alerts and add them to the new NagiosXI system
- 45. Phase IIIImplement Global MonitoringAdd monitors for existing international systems Add monitors using JMX to monitor Java serversNagios Remote Process Execution (NRPE) to monitor remotelyRemote Monitoring for Windows Servers (NS Client++)Implement notification and escalation of alertsAdd monitors for critical business functions
- 46. Phase III continued…Corporate EnhancementsRequest recurring down time enhancement from Ethan GalstadAutomate refresh of NagiosXI standby systemBuild Network MapRetire Windows SiteScopeAdd monitors for phone systems Add monitors to data center (UPS, Temperature, Humidity)Integrate to SAP Tidal monitoring tool
- 47. Phase III continued…BusinessBusiness review and approve SLA (using business terms)Monitor both the Business Functions and the individual point devices that provide the Business FunctionFollow the Sun with Eyes on Glass.Training How to setup alertsHow to receive alertsHow to report on performance graphsCreate a new Dashboard for HelpDesk and International IT Staff
- 48. Inventory of Monitor Checks
- 53. Data Centers in 7 Countries
- 55. IT Operations
- 56. IT Team Managers
- 57. SummaryMoM ~ Manager of ManagersAllow specialized toolsTool Requirements, enough but not allOwnership for implementation, sharedHandle alerts, lowest level in organizationChoose Nagios monitoring method
- 58. Tips, Tricks & DemosNagios XI Large Implementation Day 3, 2:00 Track 3 (Nate Broderick)3 DemosPerformance challenges and solutionsIntegrating monitoring solutions OracleMigrating from BAC & FoglightCustomizationGraphing, and more.
Editor's Notes
- #2: Isn’t Great to Be Here at the 1st Nagios World Conference, Ethan has done a great job with Nagios!
- #3: I would like to introduce some of our Nu Skin folks here that work on Nagios.Nate Broderick Nagios Systems EngineerScott McWhorter Production Support
- #4: Direct Selling - Lotions and Potions (or supplements), also Nourish the children
- #8: We tried for 10 years…
- #14: Everything = Critical Errors, Warnings, Information, False alerts
- #15: Remember there are lots of emails, critical emails buried in with the rest.
- #16: Decide if they think this is a critical problem, often a junior person trying to decided
- #19: Later I show which of these we did in Nagios and how.