Linux Debugging with observability Tools

Pstree command ( pstree - display a tree of processes)

Since every process (except the very first one init) in a Linux system has a parent, it sometimes makes
things easier to understand if all processes are displayed in a tree structure. You'll be glad to know there
exists a command line utility callled pstree - that displays a tree of processes.

Below is the example of the “pstree” command with few supplied options..

Process Highlighted
with PID

options used, what they mean indeed are briefly explained Below..

-n : Sort processes with the same ancestor by PID instead of by name. (Numeric sort.)

-a : Show command line arguments

-p : Show PIDs. PIDs are shown as decimal numbers in parentheses after each process name
1- Get the process ID by querying the process table

One can use the `ps` command to wean out the process to see what it is doing along with its
PID(process ID) and the command running on the particular PID and check the state of the process,
there are multiple argument available with the `ps` command..

$ ps -eo pid,user,args | grep <Process_Name>

2- Tracing Tool:
strace is a diagnostic, debugging and instructional userspace utility for Linux. It is used to monitor and
tamper with interactions between processes and the Linux kernel, which include system calls, signal
deliveries, and changes of process state.
System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems
with programs for which the source is not readily available since they do not need to be recompiled in
order to trace them. The operation of strace is made possible by the kernel feature known as ptrace.
In the simplest case, strace runs the specified command until it exits. Intercepts and records the system
calls which are called by a process and the signals which are received by a process.
The name of each system call, its arguments and its return value are printed to standard error or to the
file specified with the -o option.
Each line in the trace contains the system call name, followed by its arguments in parentheses and its
return value

a. Trace an already running process.

$ strace –p <running_process_id>

Example..

$ strace –p 1220

b. Trace Linux command System call

$ strace <command>

Example..

$ strace df –h

c. Trace Count time, calls, and errors for each system call.

$ strace -c ls > /dev/null

The -c flag to strace is used to provide a summary report of each system call including the time, number
of times called, and the number of errors recorded for each system call.

Example..

$ strace –c ls > /dev/null

Example of common command line to investigate if a process is hung:

$ strace -f -o <output file name> -p <process ID of suspect process>

Example..
 RHEL 6.7+ and RHEL 7 with strace 4.7 or above:

$ strace -fvttTyyx -s 4096 -o /tmp/strace.txt -p <process ID of suspect process>

 Older RHEL releases with strace below version 4.7. This includes all releases of RHEL 4, RHEL 5, and
RHEL 6.6 or below:

$ strace -fvttTx -s 4096 -o /tmp/strace.txt -p <process ID of suspect process>

After using the above instruction , you need to enter control-C to stop the capture of data lets say, after
5 second or whatever length of time you need to capture.

Moreover, there are couple more strace tricks which may benefit while looking for specific details to
figure out the things, below are few of them. The man page for strace provides details on all the
capabilities and options of this powerful diagnostic and profiling tool.
$ strace -i your-program  instruction pointer is at the time of the system call

$ strace -r your-program  system calls taking the longest time

$ strace -T your-program  time was spent on a particular system call

3- Resource Monitoring Tools:

Linux provides facilities to monitor the utilization of memory resources under /proc filesystem
/proc/meminfo capture the state of the physical memory

 top – process statistics

 vmstat – virtual memory statistics
 mpstat – CPU statistics
 systat – sar

How to use top to quickly list the process consuming most of the system resources?

For each listed process, the top command displays the process ID (PID), the effective user name of the
process owner (USER), the priority (PR), the nice value (NI), the amount of virtual memory the process
uses (VIRT), the amount of non-swapped physical memory the process uses (RES), the amount of shared
memory the process uses (SHR), the process status field S), the percentage of the CPU (%CPU) and
memory (%MEM) usage, the accumulated CPU time (TIME+), and the name of the executable file
(COMMAND).

For example:

However, you can the run the top command in the batch mode as below which displays a listing of the
most CPU-intensive tasks on the system. The b flag puts it into batch mode to facilitate writing the
output to disk. The c flag tells it to display the full command line for each process instead of just the
program name. The d flag tells it to delay 30 seconds between refreshes and the n flag tells it to stop
after five refreshes.

$ top -bc -d 30 -n 5

There are multiple option and argument are available with the top command which you can use to
determine the specific thing from it, likewise few tricks mentioned below which may provide you a quick
help isolating an issue. For more detail listing of arguments and options available you may use help or
man pages to refer.

$ top -p <Process_ID>  To capture and monitor the running state of a particular process

$ top -d 5  To see the disk Status every 5 second

$ top -u <user_name>  The process of a particular user

How to use vmstat (virtual memory statistics) to provide information about block IO and CPU activity
in addition to memory?
Example: vmstat 1 5

1  the values will be re-measured and reported every second

5  the values will be reported five times and then the program will stop

How to use mpstat to quickly list Multi-processor statistics, per CPU.

In the Below output you can see CPU’s statistics by per CPY basis to drill down the different values which
helps in identifying the cpu-wise information’s.

The above all commands are very useful for taking different resource statistics which are ported along
with Unix/Linux systems, However, there is a cool python based utility has been introduced called
“dstat” which provides almost all these features available in a single bundle and very handy and useful
for debugging the Linux system call and resource related issues.

Examining Linux system performance with dstat

Using the time plugin together with cpu, net, disk, system, load, proc and looking for the top consuming
cpu process ..

-top-cpu : show most expensive CPU process

This Blue marked area shows process consuming high CPU

-top-mem : show most expensive memory process

This Blue marked area shows process consuming high memory

--top-io : show most expensive I/O process

This Blue marked area shows process consuming high memory

--top-latency : show process with highest total latency (in ms)

This is Showing the process with highest latency

Below will show us the TCP statistics on the server viz how many listening , established connection etc
along with memory faults major and minor ..

Connection on time wait condition

Established connection list on the Server.

Connection is Listening condition on the Server.

This is showing overall virtual memory statistics , highlighted are Major and Minor memory Faults.

Thanks.

Karn Kumar