200.1,2-Capacity Planning
0 likes•135 views
This document provides information on monitoring Linux system resources and performance. It discusses tools like vmstat, sar, iostat for monitoring CPU usage, memory usage, I/O usage, and other metrics. It also covers Linux processes, memory management, and block device monitoring.
200.1,2-Capacity Planning
- 2. Topic 200: Capacity Planning
- 3. Object 1: Measure and Troubleshoot Resource Usage
- 5. 5 Linux performance metrics ● Processor metrics ● Memory metrics ● Block device metrics ● Network interface metrics
- 6. 6 Monitoring tools ●نیاز مورد های بسته ● sysstat ● iostat, sar, mpstat, ... ● procps ● free, uptime, vmstat, w, ...
- 7. 10 Process managementادامه ● Context switching ● Interrupt handling ● Process state (runing,rtopped,intruptible,unintruptible,zombie) fork() TASK_STOPPED TASK_UNINTERRUPTIBLE TASK_INTERRUPTIBLE TASK_RUNNING (ready) TASK_ZOMBIE exit() Processor TASK_RUNNING
- 8. 11 Processor metrics ● CPU utilization (Exceed 80%=bottleneck) ● User time, System time,Waiting, Idle time, Nice time ● Load avarage ● Runable processes (Excedd 10 time the amount of prcessors) ● Blocked (Uninterruptible: like process waiting for I/O) ● Context switch, Interrupts(software bottleneck)
- 9. 13 vmstat $ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 262008 544552 21036 392400 1 4 173 142 136 332 15 5 77 2 $ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 262008 544552 21036 392400 1 4 173 142 136 332 15 5 77 2 Process (procs) r: The number of processes waiting for runtime b: The number of processes in uninterruptable sleep Memory swpd: The amount of virtual memory used (KB) free: The amount of idle memory (KB) buff: The amount of memory used as buffers (KB) cache: The amount of memory used as cache (KB) Swap si: Amount of memory swapped from the disk (Kbps) so: Amount of memory swapped to the disk (Kbps) IO bi: Blocks sent to a block device (blocks/s) bo: Blocks received from a block device (blocks/s) System in: The number of interrupts per second, including the clock cs: The number of context switches per second Process (procs) r: The number of processes waiting for runtime b: The number of processes in uninterruptable sleep Memory swpd: The amount of virtual memory used (KB) free: The amount of idle memory (KB) buff: The amount of memory used as buffers (KB) cache: The amount of memory used as cache (KB) Swap si: Amount of memory swapped from the disk (Kbps) so: Amount of memory swapped to the disk (Kbps) IO bi: Blocks sent to a block device (blocks/s) bo: Blocks received from a block device (blocks/s) System in: The number of interrupts per second, including the clock cs: The number of context switches per second
- 10. 14 sar $ sar -q 02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 03:05:01 PM 0 566 1.83 2.01 2.16 0 $ sar -q 02:55:02 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 03:05:01 PM 0 566 1.83 2.01 2.16 0 $ sar -P ALL Average: CPU %user %nice %system %iowait %steal %idle Average: all 15.21 0.03 5.20 1.74 0.00 77.81 Average: 0 16.22 0.03 5.25 1.95 0.00 76.54 Average: 1 16.14 0.03 5.38 1.97 0.00 76.48 Average: 2 14.24 0.04 5.02 1.52 0.00 79.18 Average: 3 14.24 0.03 5.16 1.51 0.00 79.06 $ sar -P ALL Average: CPU %user %nice %system %iowait %steal %idle Average: all 15.21 0.03 5.20 1.74 0.00 77.81 Average: 0 16.22 0.03 5.25 1.95 0.00 76.54 Average: 1 16.14 0.03 5.38 1.97 0.00 76.48 Average: 2 14.24 0.04 5.02 1.52 0.00 79.18 Average: 3 14.24 0.03 5.16 1.51 0.00 79.06
- 11. 15 sar $ sar -w Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM proc/s cswch/s 02:05:01 AM 1.14 7087.09 02:15:01 AM 1.19 6955.23 02:25:01 AM 1.12 6855.21 02:35:01 AM 1.21 6311.18 02:45:01 AM 1.20 6006.53 $ sar -w Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM proc/s cswch/s 02:05:01 AM 1.14 7087.09 02:15:01 AM 1.19 6955.23 02:25:01 AM 1.12 6855.21 02:35:01 AM 1.21 6311.18 02:45:01 AM 1.20 6006.53
- 12. 16 Other command $ mpstat -P ALL Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 04:50:28 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47 0.00 0.00 78.13 $ mpstat -P ALL Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 04:50:28 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 04:50:28 PM all 14.94 0.03 4.62 1.80 0.01 0.47 0.00 0.00 78.13 $ iostat -x Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.92 0.03 5.10 1.79 0.00 78.15 $ iostat -x Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14.92 0.03 5.10 1.79 0.00 78.15
- 13. 17 Linux memory architecture ● Physical and virtual memory ● Virtual memory manager ● Page frame ● Page frame reclaiming (Swap ,...) ● Buddy system
- 14. 18 Memory metrics ● Free memory ● Swap usage ● Buffer and caches ● Slabs (kernel usage of memory) ● Active versus inactive memory ● Inactive memory is a likely candidate to be swapped out to disk
- 15. 19 free, top, vmstat $ top -b -n 1 | head -5 top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65, 0.58 Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie %Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi, 0.5 si, 0.0 st KiB Mem: 2051244 total, 1600512 used, 450732 free, 29236 buffers KiB Swap: 4954776 total, 254804 used, 4699972 free, 445900 cached $ top -b -n 1 | head -5 top - 16:54:00 up 5:39, 3 users, load average: 0.93, 0.65, 0.58 Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie %Cpu(s): 14.9 us, 4.6 sy, 0.0 ni, 78.2 id, 1.8 wa, 0.0 hi, 0.5 si, 0.0 st KiB Mem: 2051244 total, 1600512 used, 450732 free, 29236 buffers KiB Swap: 4954776 total, 254804 used, 4699972 free, 445900 cached $ vmstat -a procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free inact active si so bi bo in cs us sy id wa 0 1 254804 453648 613696 844384 1 3 155 129 309 480 15 5 78 2 $ vmstat -a procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free inact active si so bi bo in cs us sy id wa 0 1 254804 453648 613696 844384 1 3 155 129 309 480 15 5 78 2
- 16. 20 sar $ sar -r Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact 02:05:01 AM 289476 1761768 85.89 75248 547476 3017772 43.07 1141260 480772 $ sar -r Linux 3.2.0-4-amd64 (laptop) 01/29/2015 _x86_64_ (4 CPU) 01:51:22 AM LINUX RESTART 01:55:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact 02:05:01 AM 289476 1761768 85.89 75248 547476 3017772 43.07 1141260 480772
- 17. 25 Block device metrics ● Iowait ● Average queue length ● Average wait ● Transfers per second ● Blocks read/write per second ● Kilobytes per second read/write
- 18. 26 iostat $ iostat -p ALL 1 1$ iostat -p ALL 1 1 $ iostat -x 1 1 Linux 3.2.0-4-amd64 (laptop) 05/29/2014 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 8.62 0.03 2.61 2.15 0.00 86.60 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 1.44 5.36 3.35 4.37 196.43 59.67 66.38 0.25 32.64 12.89 47.76 6.00 4.63 $ iostat -x 1 1 Linux 3.2.0-4-amd64 (laptop) 05/29/2014 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 8.62 0.03 2.61 2.15 0.00 86.60 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 1.44 5.36 3.35 4.37 196.43 59.67 66.38 0.25 32.64 12.89 47.76 6.00 4.63
- 19. 27 sar $ sar -b Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM tps rtps wtps bread/s bwrtn/s 12:45:01 PM 29.74 21.93 7.81 1389.04 292.90 12:55:01 PM 5.90 1.34 4.56 32.94 97.90 01:05:01 PM 20.23 13.89 6.35 760.67 698.30 01:15:01 PM 83.81 72.50 11.31 1943.13 1598.73 01:25:01 PM 5.55 1.47 4.08 39.11 100.80 $ sar -b Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM tps rtps wtps bread/s bwrtn/s 12:45:01 PM 29.74 21.93 7.81 1389.04 292.90 12:55:01 PM 5.90 1.34 4.56 32.94 97.90 01:05:01 PM 20.23 13.89 6.35 760.67 698.30 01:15:01 PM 83.81 72.50 11.31 1943.13 1598.73 01:25:01 PM 5.55 1.47 4.08 39.11 100.80
- 20. 28 sar $ sar -d Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55 0.96 33.07 5.69 16.93 12:55:01 PM dev8-0 5.90 32.94 97.90 22.17 0.07 11.91 5.40 3.19 01:05:01 PM dev8-0 20.23 760.67 698.30 72.10 0.43 21.23 3.50 7.09 $ sar -d Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 12:45:01 PM dev8-0 29.74 1389.04 292.90 56.55 0.96 33.07 5.69 16.93 12:55:01 PM dev8-0 5.90 32.94 97.90 22.17 0.07 11.91 5.40 3.19 01:05:01 PM dev8-0 20.23 760.67 698.30 72.10 0.43 21.23 3.50 7.09 $ ls -l /dev/sda brw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda $ ls -l /dev/sda brw-rw---T 1 root disk 8, 0 Apr 23 12:34 /dev/sda
- 21. 31 Network interface metrics ● Packets received and sent, Bytes received and sent ● Collisions per second ● Packets dropped ● Overruns (ran out of buffer space) ● Errors
- 22. 32 ifconfig $ ifconfig eth0 wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76 inet addr:192.168.13.51 Bcast:192.168.13.255 Mask:255.255.255.0 inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:272597 errors:0 dropped:0 overruns:0 frame:0 TX packets:259457 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0 MiB) $ ifconfig eth0 wlan0 Link encap:Ethernet HWaddr 00:08:ca:45:d0:76 inet addr:192.168.13.51 Bcast:192.168.13.255 Mask:255.255.255.0 inet6 addr: fe80::208:caff:fe45:d076/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:272597 errors:0 dropped:0 overruns:0 frame:0 TX packets:259457 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:350358017 (334.1 MiB) TX bytes:48277958 (46.0 MiB)
- 23. 33 sar $ sar -n DEV Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 12:45:01 PM lo 0.07 0.07 0.01 0.01 0.00 0.00 0.00 12:45:01 PM wlan0 12.83 12.76 13.38 2.39 0.00 0.00 0.00 12:45:01 PM kvnet 7.97 8.28 8.10 1.10 0.00 0.00 0.00 12:45:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 $ sar -n DEV Linux 3.2.0-4-amd64 (laptop.cvak.local) 04/23/2015 _x86_64_ (4 CPU) 12:34:50 PM LINUX RESTART 12:35:04 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 12:45:01 PM lo 0.07 0.07 0.01 0.01 0.00 0.00 0.00 12:45:01 PM wlan0 12.83 12.76 13.38 2.39 0.00 0.00 0.00 12:45:01 PM kvnet 7.97 8.28 8.10 1.10 0.00 0.00 0.00 12:45:01 PM eth0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
- 24. 34 netstat # netstat -antup Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.13.51:80 192.168.13.52:53930 ESTABLISHED 27398/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53931 ESTABLISHED 27399/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53930 TIME_WAIT - tcp 0 0 192.168.13.60:80 192.168.13.52:53931 SYN_RECV 27559/apache2 # netstat -antup Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.13.51:80 192.168.13.52:53930 ESTABLISHED 27398/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53931 ESTABLISHED 27399/apache2 tcp 0 0 192.168.13.51:80 192.168.13.52:53930 TIME_WAIT - tcp 0 0 192.168.13.60:80 192.168.13.52:53931 SYN_RECV 27559/apache2 # netstat -s# netstat -s
- 25. Object 2: Predict Future Resource Needs
- 27. 37 Collectd ●نیاز مورد های بسته نصب ●هامسیر # apt-get install collectd librrds-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl libjson-perl # apt-get install collectd librrds-perl libconfig-general-perl libhtml-parser-perl libregexp-common-perl libjson-perl Config file: /etc/collectd/{collectd.conf,collection.conf} Plugins : /usr/lib/collectd Data: /var/lib/collectd Config file: /etc/collectd/{collectd.conf,collection.conf} Plugins : /usr/lib/collectd Data: /var/lib/collectd
- 28. 38 Setting Options ● Global options (Interval, FQDNLookup, …) ● Plugin options (documented on man collectd.conf )“ ” BaseDir "/var/lib/collectd" PIDFile "/run/collectd.pid" Interval 10.0 (if changed delete all rrd files) LoadPlugin cpu LoadPlugin df <LoadPlugin df> Interval 3600 </LoadPlugin> BaseDir "/var/lib/collectd" PIDFile "/run/collectd.pid" Interval 10.0 (if changed delete all rrd files) LoadPlugin cpu LoadPlugin df <LoadPlugin df> Interval 3600 </LoadPlugin>
- 29. 39 Plugins ● Plugins ● Input (read) Plugins (CPU, Memory) ● Output (write) Plugins (RRDtool,CSV) ● Read and Write Plugins (Network) ● Log Plugins (LogFile,Syslog) # vi /etc/collectd/collectd.conf ... LoadPlugin apache LoadPlugin cpu LoadPlugin df LoadPlugin network LoadPlugin interface # vi /etc/collectd/collectd.conf ... LoadPlugin apache LoadPlugin cpu LoadPlugin df LoadPlugin network LoadPlugin interface
- 30. 41 Creating Graphs ● Debian ● Collection3 # apt-get install apache2-mpm-worker (Wheezy)# apt-get install apache2-mpm-worker (Wheezy) # cp -r /usr/share/doc/collectd/examples/collection3 /var/www/# cp -r /usr/share/doc/collectd/examples/collection3 /var/www/ # apt-get install apache2 (Jessie)# apt-get install apache2 (Jessie) # a2enmod cgi # vi /etc/apache2/sites-enabled/default <Directory /var/www/collection3> Options +ExecCgi AddHandler cgi-script .cgi .pl </Directory> # a2enmod cgi # vi /etc/apache2/sites-enabled/default <Directory /var/www/collection3> Options +ExecCgi AddHandler cgi-script .cgi .pl </Directory>