Mikhail Serkov - Zabbix for HPC Cluster Support | ZabConf2016
1 like1,138 views
For the last two years I've been working in Cambridge (US) in Novartis Institute for Biomedical Research (NIBR) on a project related to a support of HPC cluster infrastructure and users. We're using Zabbix for HPC cluster monitoring (more than 1000 nodes, 10000+ cores, GPU cores, etc). In this presentation we will cover interesting use cases of Zabbix for HPC cluster, as it's not a regular infrastructure monitoring. We will talk about some challenges we have in HPC monitoring, how Zabbix helps us to work with scientists as well as present some solutions, which might be interesting for Zabbix community.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to Mikhail Serkov - Zabbix for HPC Cluster Support | ZabConf2016 (20)
![The power of linux advanced tracer [POUG18]](https://cdn.slidesharecdn.com/ss_thumbnails/thepoweroflinuxadvancedtracer-180909125236-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Zabbix (18)
Recently uploaded (20)
Mikhail Serkov - Zabbix for HPC Cluster Support | ZabConf2016
- 1. ZABBIX FOR HPC MONITORING AND SUPPORT Mikhail Serkov Delivery Manager/HPC Engineer 2016
- 2. CONFIDENTIAL 2 What’s next? HPC monitoring – differences from classic support model How do we use Zabbix AGENDA Overview of the customer infrastructure and software stack High Performance Computing – what is it about#1 #4 #5 #2 #3
- 3. CONFIDENTIAL 3 • Scientific research in Pharma area: Bioinformatics, Computational Chemistry, Drug Discovery, etc. • About 10k CPU cores used for a scientific computation. • Shared clusters - different workflows could run simultaneously within the same cluster. • About 500 different scientific tools. • Custom software ( Python, Java, R) Novartis Institute For Biomedical Research (NIBR)
- 4. CONFIDENTIAL 4 • Hundreds or even thousands of computation nodes • Grid Computing technologies and software ( SGE, UGE, SoGE, PBS, etc) • Massive parallel computation across the nodes • Strong requirements for all subsystems on hardware and software level ( storage, network, power, OS ) • No magic. Linux boxes, shell scripts on a low level ☺ Example of a job submission: HIGH PERFORMANCE COMPUTING
- 5. CONFIDENTIAL 5 OVERVIEW OF THE CUSTOMER INFRASTRUCTURE 250 GPU’s 70TB RAM 35-40KW/Rack
- 6. CONFIDENTIAL 6 • 28 CPU cores ( 2 sockets x 14 cores each ) − Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz • 200 GB RAM • 10 GB Ethernet + InfiniBand interfaces • 8 GPU cores ( 4 cards x 2 cores each ) • NFS over 10 GB Ehternet • Lustre over InfiniBand TYPICAL COMPUTATION NODE CONFIGURATION
- 7. CONFIDENTIAL 7 OVERVIEW OF SOFTWARE STACK • More than 500 of scientific tools • Bioinformatics, Computation Chemistry, Xtallography, Molecular Dynamics, etc • RHEL6.5 • Univa Grid Engine • Zabbix 2.4
- 8. CONFIDENTIAL 8 • We need information like ‘who, what, when’, not only system metrics. • Users are allowed to run whatever they want using grid scheduler on the computation nodes. • 100% CPU utilization and 100% RAM utilization for node is perfectly fine. • Node crash – not such a big deal. • Preventing global issues by using aggregated metrics. • Metrics not only for monitoring but for a performance analysis. • Users are having access to the monitoring system ( but restricted ). HPC MONITORING DIFFERENCES
- 9. CONFIDENTIAL 9 • Able to monitor of a huge systems with a lot of metrics • Flexible • Out of the box • Ability to aggregate metrics • API for a data extraction • GUI convenient for both support team and scientists • Autodiscovery • New nodes automatic configuration WHY ZABBIX?
- 10. CONFIDENTIAL 10 ZABBIX CONFIGURATION Server configuration: • 20 CPU cores ( 2 sockets x 10 cores each ) − Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz • 120 GB RAM Number of hosts: 601 Number of items: ~200k Number of triggers: ~37k DB Size: 187GB
- 11. CONFIDENTIAL 11 WHAT DO WE MONITOR Local metrics ( node level ) Global metrics ( cluster level ) All default Linux checks (LA, CPU utilization, RAM, swap, etc) - agent Meta CPU utilization – aggregation of CPU utilization of HPC nodes. Every single GPU core ( Temperature, Utilization if possible) - agent NFS global transmit/retransmit - aggregation of nodes values Every single CPU core ( Utilization, Temperature) - agent Grid specific – used/active slots, running jobs, pending jobs, top users - external scripts NFS shares availability / utilization / mount details - agent CPU/Memory oversubscription - aggregation of nodes values Slots / RAM reserved - external scripts Overloaded nodes - aggregation of HPC values HPC jobs - external scripts Pending time - external scripts ... ....
- 12. CONFIDENTIAL 12 HPC specific examples 1) Expected utilization VS Real one Every job has a resource request for number of CPUs, RAM, etc. In every moment we can compare real utilization with an expected one. If they are not close, we need to investigate if someone oversubscribing resources or overload nodes. Solution: Zabbix not only checks current system metrics, but also keeps an expected values. If they are too different we receive warning. 2) Users on a computation node Users are not restricted to SSH to any node ( debugging, tracing job in real time, interactive jobs, etc). However we should check if user has job on the node he is logged into. Solution: We have a trigger that notify us if we have anyone logged on the node with no job running. Additionally we store a list of logged in users for any single moment.
- 13. CONFIDENTIAL 13 HPC specific examples Pending time probes It is really hard to predict the pending time for any particular job in the pending list, as they all have different resource requests, and runtimes. It is not a FIFO and the pending time is always related to resources user wants to have. Solution: Zabbix runs ‘pending probes’ ( empty jobs) and checks how long does it take. This is a good indicator for queue state at the moment.
- 14. CONFIDENTIAL 14 WHAT DO WE MONITOR: GLOBAL METRICS Global cluster utilization
- 15. CONFIDENTIAL 15 WHAT DO WE MONITOR: GLOBAL METRICS RAM oversubscription
- 16. CONFIDENTIAL 16 WHAT DO WE MONITOR: GLOBAL METRICS CPU time oversubscription
- 17. CONFIDENTIAL 17 WHAT DO WE MONITOR: GLOBAL METRICS Meta CPU utilization
- 18. CONFIDENTIAL 18 WHAT DO WE MONITOR: GLOBAL METRICS Aggregated cluster status
- 19. CONFIDENTIAL 19 WHAT DO WE MONITOR: GLOBAL METRICS Storage operational metrics
- 20. CONFIDENTIAL 20 WHAT DO WE MONITOR: LOCAL METRICS
- 21. CONFIDENTIAL 21 WHAT DO WE MONITOR: LOCAL METRICS
- 22. CONFIDENTIAL 22 USER ACCESS We want to provide a limited amount of information to users. They don’t need any info about triggers and issues, but only metrics. We have patched Zabbix to remove all unnecessary data for guest access. After Before
- 23. CONFIDENTIAL 23 Benefits • Better understanding of a global issues on the cluster an reasons of why have they happened. • Great performance indicators for other infrastructure teams ( especially Storage team ) • Performance tuning of a scientific workflows. Jobs profiling. In some cases information we cat get from Zabbix is helping us to significantly improve performance of jobs. • Proactive monitoring. With Zabbix it’s easier to understand if something is not right on the cluster or with some job. In most cases we are able to prevent global cluster issues, or at least minimize an impact. • One monitoring system for clusters and HPC infrastructure. • “All in one”. Lower efforts on support/maintain monitoring system(s).
- 24. CONFIDENTIAL 24 • Tight integration with Grid HPC software. • Data analysis using external tools, but with Zabbix data source. • Create a set of CLI utilities for getting Zabbix statistics in ‘human-readable’ format. • Automation of jobs profiling using Zabbix API. WHAT’S NEXT?