Kernel_Crash_Dump_Analysis

Kernel Crash Dump Analysis 101
Buland Singh (SME)
Red Hat

Agenda:
 Basics of kernel crash dump analysis
 Initial analysis of kernel crash dump
 Initial analysis of memory subsystem
 Initial analysis of storage subsystem
 Initial analysis of network subsystem
 Automatic analysis using “pykdump”

Basics of Kernel Crash Dump
Analysis

What are the tools/data required to perform a kernel crash dump analysis ?
a)  crash utility ( /usr/bin/crash )

 Provided by "crash" package.
          Eg: crash6.0.42.el6.x86_64.rpm

 No cross platform support.
         o Use 32 bit version of crash for vmcore capture from 32 bit kernel.
         o Use 64 bit version of crash for vmcore captured from 64 bit kernel.
         o Similarly for s390, s390x and PPC.
How to install ?
 Install it using yum or rpm command.
     # yum install crash
OR
    # rpm ivh crash6.0.42.el6.x86_64.rpm
 Compile from upstream source code (not supported by Red Hat Support):
    # git clone https://github.com/crashutility/crash.git
    # cd crash/
    # make
    # make install
b) Kernel symbol file (vmlinux) of the crashed kernel
 Provided by "kerneldebuginfo" package.
          Eg: kerneldebuginfo2.6.32431.el6.x86_64.rpm

 The version and arch of kerneldebuginfo package must match with the
version of kernel from which vmcore was captured.

How to install ?
 Subscribe to "rhel6serverdebugrpms" channel.
     # yumconfigmanager enable rhel6serverdebugrpms
     # subscriptionmanager repos enable rhel6serverdebugrpms
 Install it using  yum  or debuginfoinstall command:
     # yum install kerneldebuginfo2.6.32431.el6
     # debuginfoinstall kerneldebuginfo2.6.32431.el6
OR
 Download kerneldebuginfocommon and kerneldebuginfo rpm packages
from Red Hat Customer Portal and install it using rpm command.
     # rpm ivh kerneldebuginfocommonx86_642.6.32431.el6.x86_64.rpm
     # rpm ivh kerneldebuginfo2.6.32431.el6.x86_64.rpm
How to use (vmlinux)  without installing kerneldebuginfo package on the system ?
 Extract the kerneldebuginfo rpm package using rpm2cpio command.
     # rpm2cpio kerneldebuginfo2.6.32431.el6.x86_64.rpm | cpio idv
c) A machine of same architecture as of kernel from which vmcore was
captured.
 A kernel crash dump (vmcore) captured from a x86_64 machine can only be
viewed on a x86_64 machine.
 Similarly, a kernel crash dump (vmcore) captured from a s390 machine can
only be viewed on a s390 machine.
d) kernel crash dump file (vmcore)
 Captured using kdump/diskdump/netdump/xendump/LKCD/vmss2core
mechanisms.

e) Source code of the crashed kernel (optional)
 Provided by "src" rpm package of kernel.
          Eg: kernel2.6.32431.el6.src.rpm
 The version of "src" rpm package of kernel must match with the version of
kernel from which vmcore was captured.
How to install ?
 Install it using yum or rpm command.
     # yum install kernelsrc
     # rpm ivh kernel2.6.32431.el6.src.rpm
OR
 Extract the kernelsrc rpm package using rpm2cpio command.
     # rpm2cpio kernel2.6.32431.el6.src.rpm | cpio idv

How to open a kernel crash dump (vmcore) for analysis ?
a)  Typical postmortem debugging: [ Offline mode ]
    Syntax:
     # crash /path/to/vmlinux /path/to/vmcore
 Kernel object file and memory image are supplied, respectively.
     # crash osrelease vmcore
     2.6.32431.el6.x86_64
     # crash d 1 vmcore| grep RELEASE
     OSRELEASE=2.6.32431.el6.x86_64
    # crash /var/crash/vmcore
/usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinux
b) Live memory debugging: [ Online mode ]
   Syntax:
     # crash /path/to/vmlinux
 /dev/crash used by default for live memory image.

     # crash  /usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinux
c) Live memory debugging (with vmlinux search): [ Online mode ]
   Syntax:
   # crash

 Predefined directories are searched for proper vmlinux
 Version string matched to the running kernel (/proc/version)

What are the basic commands of crash utility ?
   o sys      : Display essential system information.
   o log      : Display the kernel ring buffer log.
   o bt       : Display a kernel stack backtrace.
   o bt f    : Display all stack data contained in a frame.
   o dis      : Disassemble a function or symbol.
   o kmem i  : Displays information about the use of kernel memory.
   o kmem s  : Displays basic kmalloc() slab data.
   o ipcs     : System V IPC facilities.
    o ps       : Displays processes in the system.
   o runq     : Displays the tasks on the run queues of each cpu.
   o mount    : Displays information about the mounted filesystems.
      o files    : Display the open files of the current context.
   o dev      : Display character and block device data.
   o dev d   : Display disk I/O statistics.
   o net      : Display the system's network device list.
   o irq      : Display IRQ data.
   o mod      : Display the currentlyinstalled modules.
   o mod t   : Display modules that are "tainted".
   o exit     : Exit the crash session.
   o extend   : Dynamically loads or unloads crash extension shared
                object.
   o help     : Get help.

What is crash extension module ?
 The crash utility was designed with extensibility in mind.
 You can create shared object library files consisting of crash command
extensions, that can be dynamically linked into the crash executable during
runtime or during initialization.
How to build a crash extension (not supported by Red Hat Support) ?
   # git  clone  https://github.com/crashutility/crash.git
   # cd  crash/
   # make
   # make  install
   # cd  extensions/
   # wget  http://people.redhat.com/anderson/extensions/swap_usage.c
   # cd ..
   # make  extensions

   # ls  l  extensions/swap_usage.so
     rwxrxrx. 1 root root 54101 Jan 22 21:11 extensions/swap_usage.so
How to load a crash extension ?
Syntax:
  crash> extend   /path/to/<name>.so
  crash> extend   /crash/extensions/swap_usage.so
  ./crash/extensions/swap_usage.so: shared object loaded
  crash> pswap  h
  Usage:
    pswap [k G] [pid | taskp]
    Enter "help pswap" for details.

Initial Analysis of Kernel Crash
Dump

How to check basic system information ?
 "sys" command displays essential system information.
crash> sys
      KERNEL: /usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinux
    DUMPFILE: /var/crash/127.0.0.12016012117:45:34/vmcore
        CPUS: 4
        DATE: Thu Jan 21 17:45:30 2016
      UPTIME: 00:02:01
LOAD AVERAGE: 0.49, 0.30, 0.11
       TASKS: 264
    NODENAME: bsingh.kernel.org
     RELEASE: 2.6.32431.el6.x86_64
     VERSION: #1 SMP Sun Nov 10 22:19:54 EST 2013
     MACHINE: x86_64  (2790 Mhz)
      MEMORY: 5.8 GB
       PANIC: "SysRq : Trigger a crash"
How to check hardware information ?
 The option (i) of "sys" command dumps the Desktop Management Interface
(DMI) string data if available in the kernel.
crash> sys i
        DMI_BIOS_VENDOR: Seabios
       DMI_BIOS_VERSION: 0.5.1
          DMI_BIOS_DATE: 01/01/2007
         DMI_SYS_VENDOR: Red Hat
       DMI_PRODUCT_NAME: KVM
    DMI_PRODUCT_VERSION: RHEL 6.5.0 PC
     DMI_PRODUCT_SERIAL:
       DMI_PRODUCT_UUID: 3D51C070C307BD3A281DCA1A689C22C6
     DMI_CHASSIS_VENDOR: Red Hat
       DMI_CHASSIS_TYPE: 1
    DMI_CHASSIS_VERSION:
     DMI_CHASSIS_SERIAL:
  DMI_CHASSIS_ASSET_TAG:
     DMI_SMBIOS_VERSION: 2.4
OR
crash> log | grep DMI:
DMI: Red Hat KVM, BIOS 0.5.1 01/01/2007

How to check kernel ring buffer (dmesg) ?
 "log" or "dmesg" command dumps the kernel log_buf contents.
crash> log | tail n 38
SysRq : Trigger a crash
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff8134b6c6>] sysrq_handle_crash+0x16/0x20
PGD 140aff067 PUD 140a9f067 PMD 0
Oops: 0002 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:01.2/usb1/11/speed
CPU 2
Modules linked in: nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc bnx2fc
cnic uio fcoe 8021q libfcoe garp stp libfc llc scsi_transport_fc scsi_tgt
xt_NFQUEUE iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables ipv6 uinput microcode
virtio_balloon virtio_net snd_hda_intel snd_hda_codec snd_hwdep snd_seq
snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc i2c_piix4 i2c_core
ext4 jbd2 mbcache virtio_blk pata_acpi ata_generic ata_piix virtio_pci virtio_ring
virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Pid: 2187, comm: bash Not tainted 2.6.32431.el6.x86_64 #1 Red Hat KVM
RIP: 0010:[<ffffffff8134b6c6>]  [<ffffffff8134b6c6>] sysrq_handle_crash+0x16/0x20
RSP: 0018:ffff8801422bde18  EFLAGS: 00010096
RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
RBP: ffff8801422bde18 R08: 0000000000000000 R09: 203a207152737953
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff81b01a40 R14: 0000000000000286 R15: 0000000000000007
FS:  00007ffdf778f700(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000140815000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 2187, threadinfo ffff8801422bc000, task ffff88018d4e8ae0)
Stack:
ffff8801422bde68 ffffffff8134b982 ffff88018d4e8ae0 ffff880100000000
<d> 0000000000000300 0000000000000002 ffff880140ace3c0 00007ffdf7799000
<d> 0000000000000002 fffffffffffffffb ffff8801422bde98 ffffffff8134ba3e
Call Trace:
[<ffffffff8134b982>] __handle_sysrq+0x132/0x1a0
[<ffffffff8134ba3e>] write_sysrq_trigger+0x4e/0x50
[<ffffffff811f328e>] proc_reg_write+0x7e/0xc0
[<ffffffff81188f78>] vfs_write+0xb8/0x1a0
[<ffffffff81189871>] sys_write+0x51/0x90
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: d0 88 81 a3 1c fe 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f
1f 44 00 00 c7 05 0d 07 75 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3
55 48 89 e5 0f 1f 44 00 00 8d 47
RIP  [<ffffffff8134b6c6>] sysrq_handle_crash+0x16/0x20
RSP <ffff8801422bde18>
CR2: 0000000000000000

How to determine the panic task ?
 The option (p) of "set" command sets the context to the panic task, or back
to the crash task on a live system.
crash> set p
    PID: 2187
COMMAND: "bash"
   TASK: ffff88018d4e8ae0  [THREAD_INFO: ffff8801422bc000]
    CPU: 2
  STATE: TASK_RUNNING (SYSRQ)
How to check the backtrace of panic task ?
 "bt" command displays a kernel stack backtrace. If no arguments are given,
the stack trace of the current context will be displayed.
crash> bt
PID: 2187   TASK: ffff88018d4e8ae0  CPU: 2   COMMAND: "bash"
#0 [ffff8801422bd9e0] machine_kexec at ffffffff81038f3b
#1 [ffff8801422bda40] crash_kexec at ffffffff810c5d92
#2 [ffff8801422bdb10] oops_end at ffffffff8152b510
#3 [ffff8801422bdb40] no_context at ffffffff8104a00b
#4 [ffff8801422bdb90] __bad_area_nosemaphore at ffffffff8104a295
#5 [ffff8801422bdbe0] bad_area at ffffffff8104a3be
#6 [ffff8801422bdc10] __do_page_fault at ffffffff8104ab6f
#7 [ffff8801422bdd30] do_page_fault at ffffffff8152d45e
#8 [ffff8801422bdd60] page_fault at ffffffff8152a815
    [exception RIP: sysrq_handle_crash+22]
    RIP: ffffffff8134b6c6  RSP: ffff8801422bde18  RFLAGS: 00010096
    RAX: 0000000000000010  RBX: 0000000000000063  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000000063
    RBP: ffff8801422bde18   R8: 0000000000000000   R9: 203a207152737953
    R10: 0000000000000000  R11: 0000000000000000  R12: 0000000000000000
    R13: ffffffff81b01a40  R14: 0000000000000286  R15: 0000000000000007
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#9 [ffff8801422bde20] __handle_sysrq at ffffffff8134b982
#10 [ffff8801422bde70] write_sysrq_trigger at ffffffff8134ba3e
#11 [ffff8801422bdea0] proc_reg_write at ffffffff811f328e
#12 [ffff8801422bdef0] vfs_write at ffffffff81188f78
#13 [ffff8801422bdf30] sys_write at ffffffff81189871
#14 [ffff8801422bdf80] system_call_fastpath at ffffffff8100b072
    RIP: 0000003e2c2db560  RSP: 00007fff425f6548  RFLAGS: 00010202
    RAX: 0000000000000001  RBX: ffffffff8100b072  RCX: 00000000004b5e34
    RDX: 0000000000000002  RSI: 00007ffdf7799000  RDI: 0000000000000001
    RBP: 00007ffdf7799000   R8: 000000000000000a   R9: 00007ffdf778f700
    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000002
    R13: 0000003e2c58e7a0  R14: 0000000000000002  R15: 0000003e2c58e7a0
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

How to determine the UID and GID of a process ?
RHEL6:
 struct task_struct contains the PID of a process.
crash> struct task_struct.pid ox
struct task_struct {
  [0x4a8] pid_t pid;
}
 struct cred contains the UID and GID of a process.
crash> struct cred.uid ox
struct cred {
   [0x4] uid_t uid;
}
crash> struct cred.gid ox
struct cred {
   [0x8] gid_t gid;
}
 struct cred * can be determined from struct task_struct
crash> struct task_struct.cred ox
struct task_struct {
  [0x648] const struct cred *cred;
}
crash> set 2187
    PID: 2187
COMMAND: "bash"
    CPU: 2
crash> struct task_struct.pid ffff88018d4e8ae0
  pid = 2187
crash> struct task_struct.cred ffff88018d4e8ae0
  cred = 0xffff88014c2a8200
crash> struct cred.uid 0xffff88014c2a8200
  uid = 0
crash> struct cred.gid 0xffff88014c2a8200
  gid = 0

RHEL5:
 struct task_struct contains the PID, UID and GID of a process.
crash> ps |grep java |tail n 10
  32764      1   3  ffff8102d7a26080  UN   0.0       0      0  java
crash> struct task_struct.pid,uid,gid ffff8102d7a26080
  pid = 32764,
  uid = 29109,
  gid = 9804,

How to display parental hierarchy of a process ?
 The (p) option of "ps" command displays the parental hierarchy of selected,
or all, tasks.
crash> ps p 2187
PID: 0      TASK: ffffffff81a8d020  CPU: 0   COMMAND: "swapper"
PID: 1      TASK: ffff880192d39500  CPU: 2   COMMAND: "init"
  PID: 1865   TASK: ffff88014cf00040  CPU: 1   COMMAND: "sshd"
   PID: 1877   TASK: ffff88018d606040  CPU: 2   COMMAND: "sshd"
    PID: 2161   TASK: ffff8801420beaa0  CPU: 1   COMMAND: "bash"
How to display the child task of a process ?
 The (c) option of "ps" command displays the children of selected, or all,
tasks.
crash> ps c 2187
  (no children)
crash> ps c 2161
PID: 2161   TASK: ffff8801420beaa0  CPU: 1   COMMAND: "bash"

How to check list of open files by panic task ?
 "files" command displays the open files of the current context.
crash> files 2187
ROOT: /    CWD: /root
FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff880140ac6d40 ffff880141057500 ffff880141037d58 CHR  /dev/pts/0
  1 ffff880140ace3c0 ffff8801410738c0 ffff880141043078 REG  /proc/sysrqtrigger
254 ffff88018cc71500 ffff8801434ef240 ffff88014104c4b8 REG  /root/system_crash.sh

How to check task priority and policy ?
 "task" command can used to determine the task priority and policy.
 "task" command displays the contents of a task's task_struct and
thread_info structures.
crash> set p
    PID: 2187
COMMAND: "bash"
    CPU: 2
crash> task 2187 R policy, prio, rt_priority
  policy = 0,
  prio = 120,
  rt_priority = 0,
Scheduling policies:

#define SCHED_NORMAL            0
#define SCHED_FIFO                    1
#define SCHED_RR                        2
#define SCHED_BATCH                3
#define SCHED_IDLE                    5
OR
crash> set p
    PID: 2187
COMMAND: "bash"
    CPU: 2
crash> task_struct.policy,prio,rt_priority 0xffff88018d4e8ae0
  policy = 0
  prio = 120
  rt_priority = 0

How to check the command line arguments and environment strings of
task ?
 The option (a) of "ps" command displays the argument and environment
data for the task.
crash> ps a automount
PID: 3948   TASK: f722ee30  CPU: 0   COMMAND: "automount"
ARG: /usr/sbin/automount timeout=60 /net program /etc/auto.net
ENV: SELINUX_INIT=YES
     CONSOLE=/dev/console
     TERM=linux
     INIT_VERSION=sysvinit2.85
     PATH=/sbin:/usr/sbin:/bin:/usr/bin
     LC_MESSAGES=en_US
     RUNLEVEL=3
     runlevel=3
     PWD=/
     LANG=ja_JP.UTF8
     PREVLEVEL=N
     previous=N
     HOME=/
     SHLVL=2
     _=/usr/sbin/automount
Note: This is information is only available if userspace contents are not filtered
from kernel crash dump.

How to determine resource limits (rlimits) of a process ?
 The option (r) of "ps" command displays resource limits (rlimits) of selected,
or all, tasks.
crash> set 1
    PID: 1
COMMAND: "init"
   TASK: ffff880192d39500  [THREAD_INFO: ffff880192d3a000]
    CPU: 2
  STATE: TASK_INTERRUPTIBLE
crash> ps r 1
PID: 1      TASK: ffff880192d39500  CPU: 2   COMMAND: "init"
      RLIMIT     CURRENT       MAXIMUM
         CPU   (unlimited)   (unlimited)
       FSIZE   (unlimited)   (unlimited)
        DATA   (unlimited)   (unlimited)
       STACK    10485760     (unlimited)
        CORE        0        (unlimited)
         RSS   (unlimited)   (unlimited)
       NPROC      45331         45331
      NOFILE      1024          4096
     MEMLOCK      65536         65536
          AS   (unlimited)   (unlimited)
       LOCKS   (unlimited)   (unlimited)
  SIGPENDING      45331         45331
    MSGQUEUE     819200        819200
        NICE        0             0
      RTPRIO        0             0
      RTTIME   (unlimited)   (unlimited)

How to determine total number of tasks in different state ?
 The option (S) of "ps" command displays a summary consisting of the
number of tasks in a task state.
crash> ps S
  RU: 5
  IN: 259
How to display only user space process ?
 The option (u) of "ps" command displays only user tasks.
crash> ps u | head
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
      1      0   2  ffff880192d39500  IN   0.0   19364   1500  init
    483      1   1  ffff88018d76a040  IN   0.0   11240   1320  udevd
   1336      1   3  ffff88014cf00aa0  IN   0.0   27640    864  auditd
   1337      1   0  ffff88018e46f540  IN   0.0   27640    864  auditd
   1361      1   0  ffff88018cde6080  IN   0.0  249092   1628  rsyslogd
   1362      1   3  ffff880191be5540  IN   0.0  249092   1628  rs:main Q:Reg
   1363      1   2  ffff88018d49c080  IN   0.0  249092   1628  rsyslogd
   1364      1   3  ffff88018e796aa0  IN   0.0  249092   1628  rsyslogd
   1396      1   2  ffff88018d4d4080  IN   0.0   18976    924  rpcbind
How to display only kernel threads ?
 The option (k) of "ps" command displays only kernel threads.
crash> ps k | head
>     0      0   0  ffffffff81a8d020  RU   0.0       0      0  [swapper]
>     0      0   1  ffff880192d81540  RU   0.0       0      0  [swapper]
      0      0   2  ffff880192d8a040  RU   0.0       0      0  [swapper]
>     0      0   3  ffff880192dc2aa0  RU   0.0       0      0  [swapper]
      2      0   1  ffff880192d38aa0  IN   0.0       0      0  [kthreadd]
      3      2   0  ffff880192d38040  IN   0.0       0      0  [migration/0]
      4      2   0  ffff880192d67540  IN   0.0       0      0  [ksoftirqd/0]
      5      2   0  ffff880192d66ae0  IN   0.0       0      0  [migration/0]
      6      2   0  ffff880192d66080  IN   0.0       0      0  [watchdog/0]

How to check the total time of a process in a specific state ?
 The option (m) of "ps" command displays the timestamp into days, hours,
minutes, seconds, and milliseconds since the task was last run on a cpu.
crash> ps m | grep RU
[0 00:00:00.000] [RU]  PID: 2187   TASK: ffff88018d4e8ae0  CPU: 2   COMMAND: "bash"
[0 00:02:01.177] [RU]  PID: 0      TASK: ffffffff81a8d020  CPU: 0   COMMAND: "swapper"
[0 00:02:01.409] [RU]  PID: 0      TASK: ffff880192d81540  CPU: 1   COMMAND: "swapper"
[0 00:02:01.408] [RU]  PID: 0      TASK: ffff880192d8a040  CPU: 2   COMMAND: "swapper"
[0 00:02:01.408] [RU]  PID: 0      TASK: ffff880192dc2aa0  CPU: 3   COMMAND: "swapper"
How to check run queue of each CPU ?
 "runq" command displays the tasks on a CFS run queue:
crash>  runq
CPU 0 RUNQUEUE: ffff880028216840
  CURRENT: PID: 0      TASK: ffffffff81a8d020  COMMAND: "swapper"
  RT PRIO_ARRAY: ffff8800282169c8
     [no tasks queued]
  CFS RB_ROOT: ffff8800282168d8
     [no tasks queued]
  CURRENT: PID: 0      TASK: ffff880192d81540  COMMAND: "swapper"
     [no tasks queued]
     [no tasks queued]
  CURRENT: PID: 2187   TASK: ffff88018d4e8ae0  COMMAND: "bash"
     [no tasks queued]
     [no tasks queued]
CPU 3 RUNQUEUE: ffff8800282d6840
  CURRENT: PID: 0      TASK: ffff880192dc2aa0  COMMAND: "swapper"
  RT PRIO_ARRAY: ffff8800282d69c8
     [no tasks queued]
  CFS RB_ROOT: ffff8800282d68d8
     [no tasks queued]

How to determine the values of sysctl parameter from vmcore ?
 Check the ".data" field of desired sysctl parameter in kernel socurce sysctl.c
file.
Source File: kernel2.6.32431.el6/kernel/sysctl.c
{
                .ctl_name         = VM_SWAPPINESS,
                .procname        = "swappiness",
                .data                 = &vm_swappiness,
                .maxlen            = sizeof(vm_swappiness),
                .mode               = 0644,
                .proc_handler  = &proc_dointvec_minmax,
                .strategy          = &sysctl_intvec,
                .extra1             = &zero,
                .extra2             = &one_hundred,
        },
crash> vm_swappiness
vm_swappiness = $1 = 60
Source File: kernel2.6.32431.el6/kernel/sysctl.c
{
                .ctl_name         = VM_PANIC_ON_OOM,
                .procname        = "panic_on_oom",
                .data                 = &sysctl_panic_on_oom,
                .maxlen            = sizeof(sysctl_panic_on_oom),
                .mode               = 0644,
                .proc_handler  = &proc_dointvec,
        },
crash> sysctl_panic_on_oom
sysctl_panic_on_oom = $2 = 1

Initial Analysis of Memory
Subsystem

How to check overall memory usage on the system ?
 "kmem i" command displays general memory usage information:
crash> kmem i
                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM  1205838       4.6 GB
         FREE  1102506       4.2 GB   91% of TOTAL MEM
         USED   103332     403.6 MB    8% of TOTAL MEM
       SHARED    14484      56.6 MB    1% of TOTAL MEM
      BUFFERS     6377      24.9 MB    0% of TOTAL MEM
       CACHED    39960     156.1 MB    3% of TOTAL MEM
         SLAB    17725      69.2 MB    1% of TOTAL MEM
   TOTAL SWAP    16382        64 MB
    SWAP USED        0            0    0% of TOTAL SWAP
    SWAP FREE    16382        64 MB  100% of TOTAL SWAP
COMMIT LIMIT   616741       2.4 GB
    COMMITTED    59755     233.4 MB    9% of TOTAL LIMIT
How to check perzone memory statistics ?
 "kmem z" command displays perzone memory statistics.
crash> kmem z
NODE: 0  ZONE: 0  ADDR: ffff880000010000  NAME: "DMA"
  SIZE: 4095  PRESENT: 3831  MIN/LOW/HIGH: 42/52/63
  VM_STAT:
                NR_FREE_PAGES: 3930
             NR_INACTIVE_ANON: 0
               NR_ACTIVE_ANON: 0
             NR_INACTIVE_FILE: 0
               NR_ACTIVE_FILE: 0
               NR_UNEVICTABLE: 0
                     NR_MLOCK: 0
                NR_ANON_PAGES: 0
               NR_FILE_MAPPED: 0
                NR_FILE_PAGES: 0
                NR_FILE_DIRTY: 0
                 NR_WRITEBACK: 0
          NR_SLAB_RECLAIMABLE: 0
        NR_SLAB_UNRECLAIMABLE: 0
                 NR_PAGETABLE: 0
              NR_KERNEL_STACK: 0
              NR_UNSTABLE_NFS: 0

How to determine memory usage in userspace ?
 The "ps Gu" command can be used to determine the RSS of userspace
tasks.
crash> ps Gu | sed 's/>//g' | awk '{ total += $8 } END { printf "Total RSS
of usermode: %.02f GiBn", total/2^20 }'
Total RSS of usermode: 30.74 GiB
 Per process memory usage:
crash> ps G | sort k 8,8 n r | head
> 28882  20109   1  ffff880831e0e040  RU  92.8 65272428 32120772  rsession
>  6985   6984   2  ffff88082e934040  RU   0.0  641664   4680     coda
>  6392      1   3  ffff88082e934ab0  RU   0.0  167336   6324     perfd
   6123      1   2  ffff88082c194ab0  IN   0.1  238480  45732     splunkd
   6946   6945   1  ffff88082f804ab0  RU   0.0  833744   9188     opcmona
  27387      1   0  ffff88082d288040  RU   0.0  129952   6472     bgsagent
   6275      1   0  ffff88082f7eaab0  UN   0.0   36284   5184     scopeux
   2098   2097   3  ffff88082dbe7520  IN   0.0 1454328   5172     python
   6981   6980   1  ffff880831017520  RU   0.0  903548   3236     opcmsga
   2151      1   3  ffff88082daed520  IN   0.0  706320   2748     EracentEUAServi
 Specific process's memory usage:
crash> ps Gu rsession | tail n +2 | cut b2 | gawk '{mem += $8} END
{print "Total " mem/1048576 " GB"}'
Total 30.6334 GB

How to displays information for each configured swap device ?
 "swap" command displays information for each configured swap device.
crash> swap
SWAP_INFO_STRUCT    TYPE       SIZE       USED     PCT  PRI  FILENAME
ffff88014be681c0    FILE      65528k       0k       0%   1  /swapfile
How to determine swap memory usage of each task ?
 Load crash extension (swap_usage.so)
crash> extend swap_usage.so
./swap_usage.so: shared object loaded
 Use "pswap" command to check swap usage per task.
crash> pswap G | sort n k 2 |tail
6946     3099 opcmona
6964     3334   agtrep
30548     9362        R
30575     9377        R
30566     9378        R
30557     9427        R
6123    14284 splunkd
2098    18739   python
19406   164229 rsession
28882  8107112 rsession

How to determine memory usage in kernelspace ?
 "kmem s" command displays basic kmalloc() slab data.
crash> kmem s | grep e NAME e anon_vma_chain
CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE
ffff88043bf20b00 anon_vma_chain            48  302042720  302042818 3922634     4k
Formula:  ( OBJSIZE  * ALLOCATED ) / 2^30
crash> !bc q
scale =2
(48*302042720)/2^30
13.50

 Memory allocated to "anon_vma_chain" slab object is ~13.50 GiB.

How to check memory allocated for hugepages ?
RHEL6:
 "kmem h" command displays the address of hugepage hstate array entries,
along with their hugepage size, total and free counts, and name.
crash> kmem h
HSTATE SIZE FREE TOTAL NAME
ffffffff81fbb8c0 2MB 10 10 hugepages2048kB
RHEL5:
a) Total Number of Huge pages.
crash> p d nr_huge_pages
nr_huge_pages = $1 = 35845
crash> p d nr_huge_pages*(1<<21)
$2 = 75172413440
crash> !bc q
scale=2
75172413440/2^30
70.00
b) The number of huge pages in the pool that are not yet allocated.
crash> p d free_huge_pages
free_huge_pages = $2 = 0

How to check memory allocated to VMware ballooning driver (RHEL 6+) ?
 Determine the address of symbol balloon.

crash> sym balloon
ffffffffa002b600 (b) balloon [vmware_balloon]
 Determine the value of "size" variable using the address of symbol balloon.

crash> struct vmballoon.size 0xffffffffa002b600
$4 = 2721049
crash> !bc q
scale=2
2721049*4/2^20
10.37
2721049 pages = 2721049 x 4 = 10884196 KiB = 10.37 GiB
Note: The value of size variable is the amount of memory allocated by VMware
Ballooning driver in pages.
 Determine the value of "target" variable using the address of symbol balloon.

crash> vmballoon.target 0xffffffffa002b600
$5 = 3177311
crash> !bc q
scale=2
3177311*4/2^20
12.12
3177311 pages = 3177311 x 4 = 12709244 KiB = 12.12 GiB
Note: The value of target variable is the amount of memory needed by VMware
Ballooning driver in pages.

Initial Analysis of Storage
Subsystem

How to display disk I/O statistics ?
 "dev d" command displays disk I/O statistics.
crash> dev d
MAJOR GENDISK            NAME       REQUEST_QUEUE      TOTAL ASYNC  SYNC   DRV
    8 ffff88404ee6b400   sda        ffff88204c212338       0     0     0     0
  253 ffff88404eedac00   dm0       ffff88404bf60b68       0     0     0     0
  253 ffff88404eedc800   dm1       ffff88404bf60038       0     0     0     0
  253 ffff8820508c3c00   dm2       ffff882050444ea8       0     0     0     0
  253 ffff88204be1f800   dm3       ffff882050444378       0     0     0     0
  253 ffff88204dda4800   dm4       ffff882050458ee8       0     0     0     0
TOTAL   : Total number of allocated inprogress I/O requests
SYNC    : I/O requests that are synchronous
ASYNC  : I/O requests that are asynchronous
READ   : I/O requests that are reads (older kernels)
WRITE  : I/O requests that are writes (older kernels)
  DRV    : I/O requests that are inflight in the device driver

How to check the status of disks ( online/offline ) ?
Method: 1
 Determine struct gendisk * using "dev d" command.
crash> dev d

struct gendisk * for /dev/sda is 0xffff88404ee6b400
 Determine "private_data" from struct gendisk *
crash> gendisk.private_data ox
struct gendisk {
  [0x2c8] void *private_data;
}
crash> gendisk.private_data ffff88404ee6b400
  private_data = 0xffff88404ee6b000
 Determine struct scsi_device * from struct scsi_disk *
crash> scsi_disk.device ox
struct scsi_disk {
    [0x8] struct scsi_device *device;
}
crash> scsi_disk.device 0xffff88404ee6b000
  device = 0xffff88204c090800
 Determine sdev_state from struct scsi_device *
crash> scsi_device.sdev_state ox
struct scsi_device {
  [0x5a0] enum scsi_device_state sdev_state;
}
crash> scsi_device.sdev_state 0xffff88204c090800
  sdev_state = SDEV_RUNNING
 The sdev state are defined in include/scsi/scsi_device.h file.

enum scsi_device_state {
        SDEV_CREATED = 1,       /* device created but not added to sysfs
                                                     * Only internal commands allowed (for inq) */
        SDEV_RUNNING,            /* device properly configured
                                                    * All commands allowed */
        SDEV_CANCEL,              /* beginning to delete device
                                                    * Only error handler commands allowed */
        SDEV_DEL,                      /* device deleted
                                                    * no commands allowed */
        SDEV_QUIESCE,            /* Device quiescent.  No block commands
                                                   * will be accepted, only specials (which
                                                   * originate in the midlayer) */
        SDEV_OFFLINE,            /* Device offlined (by error handling or
                                                   * user request */
        SDEV_BLOCK,                /* Device blocked by scsi lld.  No
                                                   * scsi commands from user or midlayer
                                                   * should be issued to the scsi
                                                   * lld. */
        SDEV_CREATED_BLOCK,     /* same as above but for created devices */
};

Method: 2
 Determine struct request_queue * using "dev d" command.
crash> dev d

struct request_queue * for /dev/sda is 0xffff88204c212338
 Determine "queuedata" from struct request_queue *
crash> request_queue.queuedata ox
struct request_queue {
  [0x300] void *queuedata;
}
crash> request_queue.queuedata ffff88204c212338
  queuedata = 0xffff88204c090800
 Determine sdev_state from struct scsi_device *
crash> scsi_device.sdev_state ox
struct scsi_device {
  [0x5a0] enum scsi_device_state sdev_state;
}
crash> scsi_device.sdev_state 0xffff88204c090800
  sdev_state = SDEV_RUNNING

How to determine the default I/O scheduler set ?
 Search keyword "scheduler" in kernel ring buffer.
 Check the line which ends with "(default)".
crash> log | grep scheduler
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)

How to determine the I/O scheduler set on a specific disk ?
 Determine the struct request_queue * from "dev d" command output.
crash> dev d
  252 ffff88018cc1ec00   vda        ffff88018c460e68       0     0     0     0
struct request_queue * for /dev/sda is 0xffff88204c212338
 Determine struct elevator_queue * from struct request_queue *
crash> request_queue.elevator ox
struct request_queue {
   [0x18] struct elevator_queue *elevator;
}
crash> request_queue.elevator ffff88018c460e68
  elevator = 0xffff88018d4b1e40
 Determine struct elevator_type * from struct elevator_queue *
crash> elevator_queue.elevator_type ox
struct elevator_queue {
  [0x50] struct elevator_type *elevator_type;
}
crash> elevator_queue.elevator_type 0xffff88018d4b1e40
  elevator_type = 0xffffffff81af24e0 <iosched_cfq>
 Determine elevator_name[16] from struct elevator_type *
crash> elevator_type.elevator_name ox
struct elevator_type {
  [0xb0] char elevator_name[16];
}

crash> elevator_type.elevator_name 0xffffffff81af24e0
elevator_name = "cfq000000000000000000000000000000000000"

Initial Analysis of Network
Subsystem

How to display the system's network device list ?
 "net" command is displays the system's network device list.
crash> net
   NET_DEVICE     NAME   IP ADDRESS(ES)
ffff8801924bd020  lo     127.0.0.1
ffff88018cda7020  eth0   192.168.122.189
How to display ARP cache ?
 "net a" command dump the ARP cache.
crash> net a
NEIGHBOUR        IP ADDRESS      HW TYPE    HW ADDRESS         DEVICE  STATE
ffff88018ccf2080 192.168.122.1   ETHER      52:54:00:1a:b1:81  eth0    REACHABLE
ffff88018ccf2e80 0.0.0.0         UNKNOWN    00 00 00 00 00 00  lo      NOARP

Automatic Analysis using
“pykdump”

What is pykdump (not supported by Red Hat Support) ?
 pykdump is a python framework to access Linux dumps programmatically
from "crash".
 pykdump can be downloaded from http://sourceforge.net/projects/pykdump
          Filename: mpykdump0.8.2x86_64.tar.gz
 It provide two files:
     # tar xvf mpykdump0.8.2x86_64.tar.gz
       README
       usr/local/lib/mpykdump64.so
       usr/local/bin/crash64
How to use pykdump ?
 Load crash extension mpykdump64.so.
crash> extend  /usr/local/lib/mpykdump64.so
Setting scroll off while initializing PyKdump
/usr/local/lib/mpykdump64.so: shared object loaded
 crash extension mpykdump64.so provides two very useful commands.

  o crashinfo
  o xportshow
[ crashinfo ]
 crashinfo is a tool gather information directly on host machine.
 It do some simple automated firstpass analysis.
 It useful for hangs or performance problems.
[ xportshow ]
 xportshow is a tool to extract and print the networking transport related
information from Linux kernel dumps.

How to perform a initial analysis of kernel crash dump automatically ?
crash> crashinfo
 By default, when crashinfo is invoked without any options, it prints some
vital statistics and runs a number of tests.
 As a result of running these tests, it can print WARNING messages.
 These messages are usually (but not always) followed by more details.
How to print sysctl Information using crashinfo command ?
crash> crashinfo sysctl | head
abi.vsyscall32       1
crypto.fips_enabled  0
debug.exceptiontrace 1
debug.kprobesoptimization 0
dev.hpet.maxuserfreq 64
dev.mac_hid.mouse_button2_keycode 97
dev.mac_hid.mouse_button3_keycode 100
dev.mac_hid.mouse_button_emulation 0
dev.raid.speed_limit_max 200000
dev.raid.speed_limit_min 1000

How to print Interface Information using xportshow command ?
crash> xportshow i
====================== lo <struct net_device 0xffff881028cb5020>  ============
lo             127.0.0.1/8  mtu=65536                        LOOPBACK
  inet6 addr: ::1/128
    flags=<IFF_UP|IFF_LOOPBACK>
    features=<FRAGLIST|SG|NO_CSUM|HIGHDMA|UFO|TSO|VLAN_CHALLENGED|LLTX>
====================== eth0 <struct net_device 0xffff881020546020>  ==========
eth0     64.101.214.136/26  mtu=1500      00:50:56:b4:75:89  ETHER
  inet6 addr: fe80::250:56ff:feb4:7589/64
    flags=<IFF_UP|IFF_BROADCAST|IFF_MULTICAST>
    features=<SG|HW_VLAN_TX|HW_VLAN_FILTER|HIGHDMA|HW_CSUM|HW_VLAN_RX|TSO>
How to print sysctl information for net using xportshow command  ?
crash> xportshow sysctl | head n 5
net.ipv4.cipso_cache_bucket_size              10
net.ipv4.cipso_cache_enable                   1
net.ipv4.cipso_rbm_optfmt                     0
net.ipv4.cipso_rbm_strictvalid                1
net.ipv4.igmp_max_memberships                 20
How to print ARP & Neighbouring information using xportshow command ?
crash> xportshow arp
=== <struct neigh_table 0xffffffffa015b080> PF_INET6 nd_tbl
IP ADDRESS        HW TYPE    HW ADDRESS           DEVICE  STATE

::                LOOPBACK   00:00:00:00:00:00    lo      NOARP
=== <struct neigh_table 0xffffffff81b205c0> PF_INET arp_tbl
IP ADDRESS        HW TYPE    HW ADDRESS           DEVICE  STATE

64.101.214.135    ETHER      00:50:56:b4:63:22    eth0    REACHABLE
64.101.214.129    ETHER      00:00:0c:9f:f0:01    eth0    REACHABLE
0.0.0.0           LOOPBACK   00:00:00:00:00:00    lo      NOARP
64.101.214.132    ETHER      00:50:56:b4:5a:e4    eth0    REACHABLE
64.101.214.133    ETHER      00:50:56:b4:68:e9    eth0    REACHABLE

How to display network statistics using xportshow command ?
crash> xportshow summary
TCP Connection Info

        ESTABLISHED      1
             LISTEN     19
NAGLE disabled (TCP_NODELAY):     1
user_data set (NFS etc.):         4
UDP Connection Info

  20 UDP sockets, 0 in ESTABLISHED
user_data set (NFS etc.):         4
Unix Connection Info

        ESTABLISHED    179
              CLOSE     20
             LISTEN     25
Raw sockets info

    None
Interfaces Info

  How long ago (in seconds) interfaces trasmitted/received?
  Name     RX          TX

   lo         n/a          n/a
eth0         n/a          0.0

Kernel_Crash_Dump_Analysis

Recommended

More Related Content

What's hot (20)

Similar to Kernel_Crash_Dump_Analysis (20)

More from Buland Singh (6)

Kernel_Crash_Dump_Analysis