Linux Performance Monitoring with vmstat and iostat Commands

Keeping a Linux system running smoothly requires vigilant monitoring of its performance. Two invaluable command-line utilities for this purpose are vmstat and iostat. These tools provide crucial insights into system-wide resource utilization, helping you identify bottlenecks and optimize performance. This comprehensive guide will explore these commands in detail, demonstrating how to use them effectively for in-depth performance analysis.

Why Monitor System Performance?

Performance monitoring is essential for several reasons ?

• Identifying Bottlenecks ? Pinpoint which resources (CPU, memory, disk I/O) are causing performance slowdowns.
• Preventing Outages ? Proactively identify potential issues before they lead to system crashes or service disruptions
• Resource Optimization ? Optimize resource allocation to ensure efficient utilization and prevent waste.
• Capacity Planning ? Gather data to forecast future resource needs and plan for system upgrades.

Introducing vmstat: Virtual Memory Statistics

vmstat (virtual memory statistics) reports information about processes, memory, paging, block IO, traps, and CPU activity. It provides a snapshot of system activity or can be run continuously to monitor changes over time.

Basic vmstat Usage

Running vmstat without any arguments provides a single report since the system boot. To get continuous updates, specify a delay in seconds ?

vmstat 1

This command will output statistics every 1 second.

Interpreting vmstat Output

The output of vmstat is divided into several columns ?

procs ? Information about processes ?

• r ? Number of runnable processes (waiting for CPU).
• b ? Number of processes in uninterruptible sleep.

memory ? Information about memory usage ?

• swpd ? Amount of virtual memory used (swapped).
• free ? Amount of idle memory.
• buff ? Memory used as buffers.
• cache ? Memory used as cache.
• inact ? Inactive memory.
• active ? Active memory.

swap ? Information about swap space ?

• si ? Amount of memory swapped in from disk (/s).
• so ? Amount of memory swapped out to disk (/s).

io ? Information about block I/O ?

• bi ? Blocks received from block devices (/s).
• bo ? Blocks sent to block devices (/s).

system ? Information about system activity ?

• in ? Number of interrupts per second, including clock interrupts.
• cs ? Number of context switches per second.

cpu ? Information about CPU utilization ?

• us ? Percentage of CPU time spent running user processes.
• sy ? Percentage of CPU time spent running system processes.
• id ? Percentage of CPU time spent idle.
• wa ? Percentage of CPU time spent waiting for I/O.
• st ? Percentage of CPU time stolen from this virtual machine by the hypervisor.

Key vmstat Metrics for Performance Analysis

• r (Runnable Processes) ? A consistently high value indicates CPU contention.
• si/so (Swap In/Out) ? Non-zero values indicate memory pressure and excessive swapping, which significantly degrades performance.
• wa (I/O Wait) ? A high value suggests I/O bottlenecks.

Introducing iostat: Input/Output Statistics

iostat (input/output statistics) reports CPU utilization and disk I/O statistics. It's crucial for identifying disk-related performance issues.

Basic iostat Usage

Like vmstat, running iostat without arguments provides a single report since system boot. To get continuous updates ?

iostat 1

This command will output statistics every 1 second.

To get statistics for specific devices ?

iostat -x -d /dev/sda 1

This will show statistics for the /dev/sda device every 1 second.

Interpreting iostat Output

The output of iostat includes ?

CPU Utilization ? Similar to vmstat, it shows %user, %nice, %system, %iowait, %steal, and %idle.

Device Utilization ? For each device, it shows ?

• /s ? Read operations per second
• rkB/s ? Read kilobytes per second
• rrqm/s ? Merged read requests per second
• %rrqm ? Percentage of merged read requests
• r_await ? Average read request time
• rareq-sz ? Average read request size
• w/s ? Write operations per second
• wkB/s ? Write kilobytes per second
• wrqm/s ? Merged write requests per second
• %wrqm ? Percentage of merged write requests
• w_await ? Average write request time
• wareq-sz ? Average write request size
• d/s ? Discard operations per second
• dkB/s ? Discard kilobytes per second
• drqm/s ? Merged discard requests per second
• %drqm ? Percentage of merged discard requests
• d_await ? Average discard request time
• dareq-sz ? Average discard request size
• f/s ? Flush operations per second
• f_await ? Average flush request time
• aqu-sz ? Average queue length
• %util ? Device utilization percentage

Key iostat Metrics for Performance Analysis

• %util (Device Utilization) ? A value close to 100% indicates that the disk is saturated and is a major bottleneck.
• await (Average Wait Time) ? High values indicate long wait times for I/O requests, suggesting disk performance issues.
• svctm (Service Time) ? (Use with caution as it will be removed) ideally should be lower than await. A high value compared to await suggests a problem with the device itself.

Combining vmstat and iostat

Using vmstat and iostat together provides a comprehensive view of system performance. For example, if vmstat shows high wa (I/O wait) and iostat shows high %util for a specific disk, it confirms a disk I/O bottleneck.

Example Scenario ? If you see high r (runnable processes) in vmstat and high %user in both vmstat and iostat, it indicates CPU-bound processes. If you see high si/so in vmstat, it shows memory swapping. If you see high wa in vmstat and high %util and await in iostat, it points to disk I/O bottlenecks.

Conclusion

vmstat and iostat are essential tools for Linux performance monitoring. By understanding their output and combining their insights, you can effectively identify performance bottlenecks, optimize resource utilization, and ensure the smooth operation of your Linux systems. Regularly using these tools will significantly enhance your ability to diagnose and resolve performance issues, leading to a more stable and efficient computing environment.