How IBM's Massive POWER9 UNIX Servers Benefit from InfluxDB and Grafana Technology
0 likes168 views
The document discusses the advancements in IBM's Power9 servers and how they benefit from technologies like InfluxDB and Grafana for performance monitoring. It highlights the evolution of the monitoring tool 'nmon', which has transitioned to support JSON output and direct integration with InfluxDB. The author also emphasizes the importance of real-time data and customizable graphing capabilities in managing server performance and statistics effectively.
![19/11/2020
In 2018:
What would I do differently?
Every possible statistic
Standard format [not .csv]
Central database [not local files]
Live graphs
In 2018:
What would I do differently?
7
8](https://image.slidesharecdn.com/devopsibminfluxgrafanaandnjmon-201119171813/85/How-IBM-s-Massive-POWER9-UNIX-Servers-Benefit-from-InfluxDB-and-Grafana-Technology-4-320.jpg)
![19/11/2020
Grafana
njmond.py
JSON
njmon -e
Python
Client
Python
Client
beats
PrometheusTelegraf
Direct
Direct
JSON
New
New
InfluxDB
InfluxDB
n[ji]mon
Time-Series
Infrastructure
njmon -w
JSON
JSON
JSON
Line Protocol
Line Protocol
Boot Strap
+
InfluxDB Grafana
Both offer a Cloud Service
- Pay you bill & they run it
- Remote access to save data
- Remote access for graphing
Both in-house Enterprise
- You buy and run on your kit
- Get extra features
- Get full support
Both offer Open Source
- Free access to the code
- Free downloads pre-compiled
- For Linux – AMD64, ARM
- Also MacOS and Windows!
Both available on POWER8 +
POWER9 on Linux (RHEL &
SUSE) & AIX via our friends at
https://power-devops.com
15
16](https://image.slidesharecdn.com/devopsibminfluxgrafanaandnjmon-201119171813/85/How-IBM-s-Massive-POWER9-UNIX-Servers-Benefit-from-InfluxDB-and-Grafana-Technology-8-320.jpg)
![19/11/2020
Two recent ideas:
1. Not easy to document
measures & statistics names!
[Tried to find out how many stats from Linux statd?]
2. Capturing ad-hoc stats on Big
Production Servers
Answers: AIXpert Blog
29
30](https://image.slidesharecdn.com/devopsibminfluxgrafanaandnjmon-201119171813/85/How-IBM-s-Massive-POWER9-UNIX-Servers-Benefit-from-InfluxDB-and-Grafana-Technology-15-320.jpg)
How IBM's Massive POWER9 UNIX Servers Benefit from InfluxDB and Grafana Technology
- 1. 19/11/2020 How IBM's Massive POWER9 UNIX Servers Benefit from InfluxDB and Grafana Technology Nigel Griffiths Advanced Technology, IBM, UK - These are my personal opinions - IBM email: [email protected] Open Source: [email protected] @mr_nmon twitter http://tinyurl.com/njmon - njmon sourceforge project http://tinyurl.com/AIXpert - My 135 Blog https://www.youtube.com/user/nigelargriffiths - 215 Grafana LabsInfluxdata 300,000++ people are IBMers Benchmark Centres, Demonstrations, Services people, Cloud Offerings Very roughly • 1/3rd Software • 1/3rd Services • (technical + business) • 1/3rd Hardware (Systems) • (servers + storage) One chart on 1 2
- 2. 19/11/2020 1/3rd Hardware (Systems) • (servers + storage) • POWER (IBM chip POWER9) • OS: Linux, AIX (UNIX), IBM i • 192 CPU cores, 1536 HW threads • 64 TB memory, 64 adapters • Z (mainframe, IBM chip z15) • OS: z/OS, LinuxONE for Linux • Storage • FlashSystem, SAN, NVMe, . . . Second chart on POWER9 Servers Enterprise E950 E980 S922 S924 Scale-Out Midrange 2U or 4U 1 or 2 socket SMT=8 4 to 24 CPU cores 4 TB RAM 4U 4 socket SMT=8 16 to 48 CPU cores 16 TB RAM 7U to 22U 16 socket SMT=8 192 CPU cores (1536 programs running at the same time) 64 TB RAM 3 4
- 3. 19/11/2020 My claim to fame? Started 25 years ago nmon Nigel’s Monitor OS performance data On screen or CSV file Various graphing tool For AIX and Linux (any HW) nmon for AIX now part of AIX nmon for Linux open source 1,040,108+ downloads (today) My claim to fame? Started 25 years ago nmon Nigel’s Monitor OS performance data On screen or CSV file Various graphing tool For AIX and Linux (any HW) nmon for AIX now part of AIX nmon for Linux open source 1,040,108+ downloads (today) Things have changed since starting nmon - CPUs x 200,000 faster - RAM x 1 million larger - Network x 10,000 rate - Disks, SSD & NVMe - x 500,000 larger - x 10,000 faster - nmon file format = quirky & !standard 5 6
- 4. 19/11/2020 In 2018: What would I do differently? Every possible statistic Standard format [not .csv] Central database [not local files] Live graphs In 2018: What would I do differently? 7 8
- 5. 19/11/2020 Every possible statistic DONE Standard format: JSON + LP Central database: InfluxDB Live graphs: Grafana In 2018: What would I do differently? Every possible statistic DONE Standard format: JSON + LP Central database: InfluxDB Live graphs: Grafana JSON elastic & Splunk LP telegraf Prometheus In 2018: What would I do differently? 9 10
- 6. 19/11/2020 In 2020: njmon = JSON output to njmond.py central daemon nimon = InfluxDB Line Protocol direct to InfluxDB What to know more? http://nmon.sourceforge.net/njmon In 2020: Improved handling of JSON data Continues as JSON popular useful format, especially Python But added: InfluxDB Line Protocol for direct nimon agent to remote InfluxDB Wow!! Every release is like Xmas we get new toys (graphs) - Even a webpage with samples Lets talk about Grafana! 11 12
- 7. 19/11/2020 1 2 3 1. My logo = cool 2. Donut graph, yum 3. Dark mode: Helps you sleep at the desk! 4. LED graphic equaliser: draws attention to red stats 5. Button single stat and graph: high density 6. Blue Ridge Mountain range graph 7. Carpet graph – see later 4 5 6 Lets talk about Grafana! Open Source from IBMers So AIX benefits from the latest Time-Series database & graph engines from Nigel “Mr nmon” Griffiths Stats: CPU RAM Disks Paging Volume Groups Logical Volumes Networks Adapters Kernel stats Tapes Uptime User count AIO File systems System Calls Processes NFS GPFS Spectrum Scale VIOS virtual disks VIOS SEA VIOS virtual networks VIOS SSP Linux NVIDIA GPUs AIX rPerf Recent updates: - New faster centralized collector - New direct to InfluxDB = nimon - New YouTube videos for Sys Admins - New Grafana graph templates See https://tinyurl.com/njmon Very simple endpoint install InfluxDB and Grafana install in 10 minutes Grafana starter dashboards but prime value is creating any graph you want in seconds JSON output for Elastic (ELK) & Splunk Line Protocol for InfluxDB & Prometheus End-points with njmon 13 14
- 8. 19/11/2020 Grafana njmond.py JSON njmon -e Python Client Python Client beats PrometheusTelegraf Direct Direct JSON New New InfluxDB InfluxDB n[ji]mon Time-Series Infrastructure njmon -w JSON JSON JSON Line Protocol Line Protocol Boot Strap + InfluxDB Grafana Both offer a Cloud Service - Pay you bill & they run it - Remote access to save data - Remote access for graphing Both in-house Enterprise - You buy and run on your kit - Get extra features - Get full support Both offer Open Source - Free access to the code - Free downloads pre-compiled - For Linux – AMD64, ARM - Also MacOS and Windows! Both available on POWER8 + POWER9 on Linux (RHEL & SUSE) & AIX via our friends at https://power-devops.com 15 16
- 9. 19/11/2020 Boot Strap + InfluxDB Grafana 1 Install is very quick 1 minute download 6 minute install 3 minute setup + firewall + start up Just take the defaults Influx CLI: create database njmon Boot Strap + InfluxDB Grafana VIOS 2.2.6 VIOS 3.1.0 AIX 6.1 AIX 7.1 AIX 7.2 Ubuntu 18/20 SLES 12/15 RHEL 7/8 Each end-point needs an agent - Single small binary + manual pages - “ninstall” script 1 2 Install is very quick 1 minute download 6 minute install 3 minute setup + firewall + start up Just take the defaults Influx CLI: create database njmon 17 18
- 10. 19/11/2020 Boot Strap + InfluxDB Grafana VIOS 2.2.6 VIOS 3.1.0 AIX 6.1 AIX 7.1 AIX 7.2 Ubuntu 18/20 SLES 12/15 RHEL 7/8 Each end-point needs an agent - Single small binary + manual pages - “ninstall” script Each end-point: add a crontab entry 0 * * * * /usr/lbin/nimon -c 60 -k -i influx -p 8086 1 2 3 Install is very quick 1 minute download 6 minute install 3 minute setup + firewall + start up Just take the defaults Influx CLI: create database njmon Boot Strap + InfluxDB Grafana VIOS 2.2.6 VIOS 3.1.0 AIX 6.1 AIX 7.1 AIX 7.2 Ubuntu 18/20 SLES 12/15 RHEL 7/8 Each end-point needs an agent - Single small binary + manual pages - “ninstall” script Each end-point: add a crontab entry 0 * * * * /usr/lbin/nimon -c 60 -k -i influx -p 8086 Access Grafana via a browser Settings: add influx/njmon datasource From https://grafana.com/dashboards Import njmon AIX & Linux dashboards Enjoy 1 2 3 4 Install is very quick 1 minute download 6 minute install 3 minute setup + firewall + start up Just take the defaults Influx CLI: create database njmon 19 20
- 11. 19/11/2020 Any one heard of the Dolly Parton curve? Any one heard of the Dolly Parton curve? TIME CPUBUSY PMPMAM Lunch AM AfternoonMorning Batch 100% 21 22
- 12. 19/11/2020 Any one heard of the Dolly Parton curve? Three Crunch points TIME CPUBUSY PMPMAM Lunch AM AfternoonMorning Batch 100% Any one heard of the Dolly Parton curve? Three Crunch points TIME CPUBUSY PMPMAM Lunch AM AfternoonMorning Batch 100% Problems: Averaging the whole day hides the three crunch points Periodic over a day and over a week (typical busier on Friday) Periodic over a month (end of month extra reporting) and end of year! Batch overrun times 23 24
- 13. 19/11/2020 Heat map for whole days using the Grafana Carpet Plugin This is a excellent way to determining the busy day + busy hours = first step for trend forecasting WeekWeekWeek Heat map for whole days using the Grafana Carpet Plugin This is a excellent way to determining the busy day + busy hours = first step for trend forecasting Heat Map Warning: There are always red parts! WeekWeekWeek Interesting Peaks 8 to 10 am & 2 pm Tuesday to Friday Busy day is Thursday 25 26
- 14. 19/11/2020 My to do list: Work out how to graph CPU on successive Fridays 8 am to 10 pm Batch overrun can be handled with alerts but still need trending Ideas to [email protected] Could be done in: InfluxDB “flux” or Grafana Alerts Some ideas Fri Fri Fri Fri Friday (1) Remove the weeds (2) One graph with overlay selected time periods (3) 27 28
- 15. 19/11/2020 Two recent ideas: 1. Not easy to document measures & statistics names! [Tried to find out how many stats from Linux statd?] 2. Capturing ad-hoc stats on Big Production Servers Answers: AIXpert Blog 29 30
- 16. 19/11/2020 Grafana | CPU | Memory | Disks | Network | Kernel | Processes InfluxDB Measure for AIX and Linux Saving other statistics to the same njmon database. If you can get the data via a script, you can send it on with the same njmon tags in 1/100th of a second. Then graph OS stats & your stats at the same time. Measure Statistics RDBMS script: measure* -g rdbms -G commits=986.34,rollbacks=23.1,hitratio=99.3 Sales script: measure* -g sales -G itemsold=32984,avgcost=79.99,profit=-0.003 Users script: measure* -g user -G online=65389,online_mins=184,click_pm=18.2 IT-tasks times script: measure* -g tasks -G dataload=47_min,backupmin=124,batch_min=84 * Also need InfluxDB: hostname + port & Influx-DB-name Pi Returning temp of Zero Pi fell off Network Effect of outside air temperature rising to 32C Raspberry Pi 3 MicroSD card With five temperature probes 31 32
- 17. 19/11/2020 njmon Graphing LAB + InfluxDB & Grafana 7.1 - Wednesday Nigel GriffithsTechnical Staff Member [email protected] @mr_nmon & on LinkedIn https://www.youtube.com/nigelargriffiths http://tinyurl.com/AIXpert Cloud VM’s for AIX provided by IBM TechU Oct 2020 | Virtual YouTube:- https://youtu.be/XKs5dKGuFe8 If you want to know more . . . Project Website https://tinyurl.com/njmon AIXpert Blog Articles https://www.ibm.com/support /pages/aixpert-blog-nigel- griffiths-mrnmon YouTube Videos - Details on the Next slide AIX Performance Tuning Lab Monitoring Students 33 34
- 18. 19/11/2020 https://www.youtube.com/user/nigelargriffiths https://www.youtube.com/watch?v=wN5GNc9HH7Y&list=PLKQlFnmiWVydb5QdX2wz9iRfJkuuB2ec1 13 videos ~16,000 Views up to Oct 2020 ~3 hours Now a 13 part YouTube playlist njmon + InfluxDB + Grafana for monitoring AIX & Linux Performance data Summary: nmon not going away • On screen or data capture • Stable down stream infrastructure • Very popular & part of AIX njmon for new age online tooling • Lightweight single binary agent coded in C • Loads more stats • Real-time, data stream, flexible • Python’s JSON parser = fast and cool • Or direct to InfluxDB • AIX vast array of perfstat stats + VIOS stats • Linux nmon & njmon code synergy • 100’s of new stats including GPU & GPFS email [email protected] Questions https://tinyurl.com/njmon https://www.linkedin.com/in/nigel-griffiths 35 36
- 19. 19/11/2020 37 38
- 20. 19/11/2020 39 40
- 21. 19/11/2020 41