SlideShare a Scribd company logo
© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Kernel Debugging & Profiling
2© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
What to Expect?
Kernel Debugging Tools & Techniques
Kernel Profiling Tools & Mechanism
Kernel Testing Ways
3© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Debugging by Printing
printk & dmesg
int klogctl(int cmd, char *bufp, int len);
syslogd
#include <syslog.h>
void openlog(char *ident, int option, int facility);
void syslog ( int priority, char *format, …);
void closelog( void );
ident – typically short program name
option - LOG_CONS, LOG_PERROR, LOG_PID, …
facility - LOG_AUTHPRIV, LOG_CRON, LOG_DAEMON, LOG_USER, ...
priority - LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR,
LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG
klogd – bridge between kernel ring buffer and syslogd
4© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Debugging by Querying
Through Kernel Windows
/proc
Process related stuff
General system information: cpuinfo, meminfo, …
Drivers related info: devices, iomem, interrupts, ...
/sys (Utilities: sysfsutils, sysdiag)
Buses, Classes, Devices, ...
Using ioctl
Through specific drivers
Using custom commands
5© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Crash Dumping & Analysis
Oops Analysis
CONFIG_BUG, CONFIG_KALLSYMS
Call Trace
cat /proc/kallsyms
objdump -d | -S <obj | exe>
Crash Dumping (using 2 kernel images – could be same)
System Kernel: CONFIG_KEXEC, CONFIG_DEBUG_INFO
Dump-capture Kernel: CONFIG_CRASH_DUMP, CONFIG_PROC_VMCORE, Other
Architecture specific CONFIG_s
Boot System Kernel with crashkernel=size@offset, or crashkernel=range:size
Overlay Dump-capture Kernel using kexec
After Crash (Simulated using SysRq+c, Or real)
System reboots w/ Dump-capture Kernel
Collect the dump using kdump, Or simply cp /proc/vmcore <dump file>
Analyse using <gdb | crash> vmlinux <dump file>
6© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Kernel Hacking Related Options
CONFIG_DEBUG_INFO (gcc -g)
CONFIG_DEBUG_FS
CONFIG_DYNAMIC_DEBUG
CONFIG_DEBUG_STACKOVERFLOW
CONFIG_LOCKUP_DETECTOR
CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, ...
CONFIG_MAGIC_SYSRQ
CONFIG_EARLY_PRINTK
CONFIG_DEBUG_SLAB, CONFIG_DEBUG_VM, ...
CONFIG_DEBUG_STACK_USAGE
CONFIG_PANIC_ON_OOPS
CONFIG_PANIC_ON_OOPS_VALUE
7© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Kernel Debuggers
Kernel Debugger (Frontend): kdb
CONFIG_DEBUG_INFO, CONFIG_MAGIC_SYSRQ
CONFIG_KGDB, CONFIG_KGDB_KDB,
CONFIG_KGDB_SERIAL_CONSOLE
Basic Operations over Serial Console
Remote Debugging using
kgdb (Kernel Debug Server)
Remote connection & debug through gdb vmlinux
gdb <kernel src>/vmlinux /proc/kcore
Only to gather current kernel debug information
8© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Miscellaneous Tools & Techniques
Early Init Debugs
CONFIG_SERIAL_EARLYCON
CONFIG_EARLY_PRINTK (direct on h/w)
Light & Sound
JTAG
Hardware Protocol Debug
Hardware Protocol Analyzers
Network Debugging
Driver level: ethtool [options] <dev i/f>
Packet level: tcpdump [options, e.g. -vv] -i <dev i/f>
9© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Kernel Probes
kprobes → CONFIG_KPROBES
jprobes → Specialized Kprobes
For probing function entry points
kretprobes → Return Kprobes
For probing function exit points
10© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Tracing
Single Process Tracing using strace
Traces the system calls made by an application
Command: strace [options] <application>
Excellent options to tune to get only desired output
Linux Trace Toolkit
Patch @ http://www.opersys.com/LTT
Core Module
Code using Trace Services
Daemon: tracedaemon
Utilities: tracereader, tracevisualizer
Function Tracer
CONFIG_FUNCTION_TRACER & co
/sys/kernel/debug/tracing
11© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Profiling & Code Coverage
System Profiling
From /proc using LTTng
Kernel Profiling: Oprofile
CONFIG_PROFILING, CONFIG_OPROFILE, CONFIG_APIC
Daemon: oprofiled
Utilities: opcontrol, opreport, op_help, ...
Kernel Performance Profiling using perf
CONFIG_PERF_EVENTS & co
stat, list, top, record, report, ...
Kernel Code Coverage:
CONFIG_GCOV_KERNEL, CONFIG_GCOV_PROFILE_ALL
12© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Testing Possibilities
Linux Test Project (LTP)
Hosted @ http://ltp.sourceforge.net
Suite of around 3000 tests
Exercises various parts of the Kernel
Mostly automated, except net & storage ones
Command: runltp -p -l logfile
User Mode Linux (UML)
Hosted @ http://user-mode-linux.sourceforge.net
A Kernel Instance as a User Mode Process
Kernel Debugging without “oops”ing
Good for testing & experimenting with unstable kernels
13© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
What all have we learnt?
Kernel Debugging Tools & Techniques
Debugging by Printing & Querying
Crash Dumping & Analysis
Kernel Hacking Options
Kernel Debuggers
Miscellaneous Tools & Techniques
Kernel Probes
Tracing
Kernel Profiling Tools & Mechanism
Kernel Testing Ways
14© 2010-16 SysPlay Workshops <workshop@sysplay.in>
All Rights Reserved.
Any Queries?

More Related Content

What's hot (20)

I2c drivers
I2c driversI2c drivers
I2c drivers
pradeep_tewani
 
Linux Ethernet device driver
Linux Ethernet device driverLinux Ethernet device driver
Linux Ethernet device driver
艾鍗科技
 
Linux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKBLinux Kernel Booting Process (1) - For NLKB
Linux Kernel Booting Process (1) - For NLKB
shimosawa
 
Introduction to yocto
Introduction to yoctoIntroduction to yocto
Introduction to yocto
Alex Gonzalez
 
Bootloaders
BootloadersBootloaders
Bootloaders
Anil Kumar Pugalia
 
Linux PCI device driver
Linux PCI device driverLinux PCI device driver
Linux PCI device driver
艾鍗科技
 
Embedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernelEmbedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernel
Emertxe Information Technologies Pvt Ltd
 
Block Drivers
Block DriversBlock Drivers
Block Drivers
Anil Kumar Pugalia
 
Yocto Project introduction
Yocto Project introductionYocto Project introduction
Yocto Project introduction
Yi-Hsiu Hsu
 
Platform Drivers
Platform DriversPlatform Drivers
Platform Drivers
SysPlay eLearning Academy for You
 
Linux Internals - Kernel/Core
Linux Internals - Kernel/CoreLinux Internals - Kernel/Core
Linux Internals - Kernel/Core
Shay Cohen
 
I2C Drivers
I2C DriversI2C Drivers
I2C Drivers
SysPlay eLearning Academy for You
 
Linux Initialization Process (2)
Linux Initialization Process (2)Linux Initialization Process (2)
Linux Initialization Process (2)
shimosawa
 
Toolchain
ToolchainToolchain
Toolchain
Anil Kumar Pugalia
 
U boot-boot-flow
U boot-boot-flowU boot-boot-flow
U boot-boot-flow
BabuSubashChandar Chandra Mohan
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
Adrien Mahieux
 
Embedded Linux BSP Training (Intro)
Embedded Linux BSP Training (Intro)Embedded Linux BSP Training (Intro)
Embedded Linux BSP Training (Intro)
RuggedBoardGroup
 
Bootloaders (U-Boot)
Bootloaders (U-Boot) Bootloaders (U-Boot)
Bootloaders (U-Boot)
Omkar Rane
 
Qnx os
Qnx os Qnx os
Qnx os
Student
 
Introduction Linux Device Drivers
Introduction Linux Device DriversIntroduction Linux Device Drivers
Introduction Linux Device Drivers
NEEVEE Technologies
 

Viewers also liked (16)

References
ReferencesReferences
References
Anil Kumar Pugalia
 
gcc and friends
gcc and friendsgcc and friends
gcc and friends
Anil Kumar Pugalia
 
Embedded C
Embedded CEmbedded C
Embedded C
Anil Kumar Pugalia
 
Introduction to Linux Drivers
Introduction to Linux DriversIntroduction to Linux Drivers
Introduction to Linux Drivers
Anil Kumar Pugalia
 
Network Drivers
Network DriversNetwork Drivers
Network Drivers
Anil Kumar Pugalia
 
File System Modules
File System ModulesFile System Modules
File System Modules
Anil Kumar Pugalia
 
Serial Drivers
Serial DriversSerial Drivers
Serial Drivers
SysPlay eLearning Academy for You
 
PCI Drivers
PCI DriversPCI Drivers
PCI Drivers
Anil Kumar Pugalia
 
Interrupts
InterruptsInterrupts
Interrupts
Anil Kumar Pugalia
 
SPI Drivers
SPI DriversSPI Drivers
SPI Drivers
SysPlay eLearning Academy for You
 
USB Drivers
USB DriversUSB Drivers
USB Drivers
Anil Kumar Pugalia
 
BeagleBone Black Bootloaders
BeagleBone Black BootloadersBeagleBone Black Bootloaders
BeagleBone Black Bootloaders
SysPlay eLearning Academy for You
 
Linux Porting
Linux PortingLinux Porting
Linux Porting
Anil Kumar Pugalia
 
BeagleBoard-xM Bootloaders
BeagleBoard-xM BootloadersBeagleBoard-xM Bootloaders
BeagleBoard-xM Bootloaders
SysPlay eLearning Academy for You
 
Character Drivers
Character DriversCharacter Drivers
Character Drivers
Anil Kumar Pugalia
 
File Systems
File SystemsFile Systems
File Systems
Anil Kumar Pugalia
 

Similar to Kernel Debugging & Profiling (20)

Kernel Debugging & Profiling
Kernel Debugging & ProfilingKernel Debugging & Profiling
Kernel Debugging & Profiling
Anil Kumar Pugalia
 
Linux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingLinux User Space Debugging & Profiling
Linux User Space Debugging & Profiling
Anil Kumar Pugalia
 
Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
Jarod Wang
 
Linux Kernel Debugging
Linux Kernel DebuggingLinux Kernel Debugging
Linux Kernel Debugging
GlobalLogic Ukraine
 
Sysdig
SysdigSysdig
Sysdig
gnosek
 
Android memory analysis Debug slides.pdf
Android memory analysis Debug slides.pdfAndroid memory analysis Debug slides.pdf
Android memory analysis Debug slides.pdf
VishalKumarJha10
 
Performance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux KernelPerformance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux Kernel
lcplcp1
 
Debug generic process
Debug generic processDebug generic process
Debug generic process
Vipin Varghese
 
Project ACRN system debug
Project ACRN system debugProject ACRN system debug
Project ACRN system debug
Project ACRN
 
Mobile Hacking using Linux Drivers
Mobile Hacking using Linux DriversMobile Hacking using Linux Drivers
Mobile Hacking using Linux Drivers
Anil Kumar Pugalia
 
Introduction to Linux
Introduction to LinuxIntroduction to Linux
Introduction to Linux
Anil Kumar Pugalia
 
Linux Internals Part - 3
Linux Internals Part - 3Linux Internals Part - 3
Linux Internals Part - 3
SysPlay eLearning Academy for You
 
Introduction to Linux
Introduction to LinuxIntroduction to Linux
Introduction to Linux
Anil Kumar Pugalia
 
Linux Kernel Overview
Linux Kernel OverviewLinux Kernel Overview
Linux Kernel Overview
Anil Kumar Pugalia
 
ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux Awareness
Peter Griffin
 
Forensic artifacts in modern linux systems
Forensic artifacts in modern linux systemsForensic artifacts in modern linux systems
Forensic artifacts in modern linux systems
Gol D Roger
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
libfetion
 
LAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel AwarenessLAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel Awareness
Linaro
 
LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness
Peter Griffin
 
Linux kernel bug hunting
Linux kernel bug huntingLinux kernel bug hunting
Linux kernel bug hunting
Andrea Righi
 
Linux User Space Debugging & Profiling
Linux User Space Debugging & ProfilingLinux User Space Debugging & Profiling
Linux User Space Debugging & Profiling
Anil Kumar Pugalia
 
Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0Solaris Kernel Debugging V1.0
Solaris Kernel Debugging V1.0
Jarod Wang
 
Sysdig
SysdigSysdig
Sysdig
gnosek
 
Android memory analysis Debug slides.pdf
Android memory analysis Debug slides.pdfAndroid memory analysis Debug slides.pdf
Android memory analysis Debug slides.pdf
VishalKumarJha10
 
Performance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux KernelPerformance Analysis Tools for Linux Kernel
Performance Analysis Tools for Linux Kernel
lcplcp1
 
Project ACRN system debug
Project ACRN system debugProject ACRN system debug
Project ACRN system debug
Project ACRN
 
Mobile Hacking using Linux Drivers
Mobile Hacking using Linux DriversMobile Hacking using Linux Drivers
Mobile Hacking using Linux Drivers
Anil Kumar Pugalia
 
ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux Awareness
Peter Griffin
 
Forensic artifacts in modern linux systems
Forensic artifacts in modern linux systemsForensic artifacts in modern linux systems
Forensic artifacts in modern linux systems
Gol D Roger
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
libfetion
 
LAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel AwarenessLAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel Awareness
Linaro
 
LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness
Peter Griffin
 
Linux kernel bug hunting
Linux kernel bug huntingLinux kernel bug hunting
Linux kernel bug hunting
Andrea Righi
 

More from Anil Kumar Pugalia (20)

File System Modules
File System ModulesFile System Modules
File System Modules
Anil Kumar Pugalia
 
Processes
ProcessesProcesses
Processes
Anil Kumar Pugalia
 
Embedded Software Design
Embedded Software DesignEmbedded Software Design
Embedded Software Design
Anil Kumar Pugalia
 
Playing with R L C Circuits
Playing with R L C CircuitsPlaying with R L C Circuits
Playing with R L C Circuits
Anil Kumar Pugalia
 
Audio Drivers
Audio DriversAudio Drivers
Audio Drivers
Anil Kumar Pugalia
 
Video Drivers
Video DriversVideo Drivers
Video Drivers
Anil Kumar Pugalia
 
Shell Scripting
Shell ScriptingShell Scripting
Shell Scripting
Anil Kumar Pugalia
 
Functional Programming with LISP
Functional Programming with LISPFunctional Programming with LISP
Functional Programming with LISP
Anil Kumar Pugalia
 
Power of vi
Power of viPower of vi
Power of vi
Anil Kumar Pugalia
 
"make" system
"make" system"make" system
"make" system
Anil Kumar Pugalia
 
Hardware Design for Software Hackers
Hardware Design for Software HackersHardware Design for Software Hackers
Hardware Design for Software Hackers
Anil Kumar Pugalia
 
RPM Building
RPM BuildingRPM Building
RPM Building
Anil Kumar Pugalia
 
Linux Network Management
Linux Network ManagementLinux Network Management
Linux Network Management
Anil Kumar Pugalia
 
System Calls
System CallsSystem Calls
System Calls
Anil Kumar Pugalia
 
Timers
TimersTimers
Timers
Anil Kumar Pugalia
 
Threads
ThreadsThreads
Threads
Anil Kumar Pugalia
 
Synchronization
SynchronizationSynchronization
Synchronization
Anil Kumar Pugalia
 
Processes
ProcessesProcesses
Processes
Anil Kumar Pugalia
 
Signals
SignalsSignals
Signals
Anil Kumar Pugalia
 
Linux Memory Management
Linux Memory ManagementLinux Memory Management
Linux Memory Management
Anil Kumar Pugalia
 

Recently uploaded (20)

Gihbli AI and Geo sitution |use/misuse of Ai Technology
Gihbli AI and Geo sitution |use/misuse of Ai TechnologyGihbli AI and Geo sitution |use/misuse of Ai Technology
Gihbli AI and Geo sitution |use/misuse of Ai Technology
zainkhurram1111
 
Offshore IT Support: Balancing In-House and Offshore Help Desk Technicians
Offshore IT Support: Balancing In-House and Offshore Help Desk TechniciansOffshore IT Support: Balancing In-House and Offshore Help Desk Technicians
Offshore IT Support: Balancing In-House and Offshore Help Desk Technicians
john823664
 
Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 ProfessioMaster tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
Kari Kakkonen
 
UiPath Community Zurich: Release Management and Build Pipelines
UiPath Community Zurich: Release Management and Build PipelinesUiPath Community Zurich: Release Management and Build Pipelines
UiPath Community Zurich: Release Management and Build Pipelines
UiPathCommunity
 
Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)
Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)
Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)
Eugene Fidelin
 
A Comprehensive Guide on Integrating Monoova Payment Gateway
A Comprehensive Guide on Integrating Monoova Payment GatewayA Comprehensive Guide on Integrating Monoova Payment Gateway
A Comprehensive Guide on Integrating Monoova Payment Gateway
danielle hunter
 
New Ways to Reduce Database Costs with ScyllaDB
New Ways to Reduce Database Costs with ScyllaDBNew Ways to Reduce Database Costs with ScyllaDB
New Ways to Reduce Database Costs with ScyllaDB
ScyllaDB
 
Fully Open-Source Private Clouds: Freedom, Security, and Control
Fully Open-Source Private Clouds: Freedom, Security, and ControlFully Open-Source Private Clouds: Freedom, Security, and Control
Fully Open-Source Private Clouds: Freedom, Security, and Control
ShapeBlue
 
Content and eLearning Standards: Finding the Best Fit for Your-Training
Content and eLearning Standards: Finding the Best Fit for Your-TrainingContent and eLearning Standards: Finding the Best Fit for Your-Training
Content and eLearning Standards: Finding the Best Fit for Your-Training
Rustici Software
 
Cyber Security Legal Framework in Nepal.pptx
Cyber Security Legal Framework in Nepal.pptxCyber Security Legal Framework in Nepal.pptx
Cyber Security Legal Framework in Nepal.pptx
Ghimire B.R.
 
TrustArc Webinar: Mastering Privacy Contracting
TrustArc Webinar: Mastering Privacy ContractingTrustArc Webinar: Mastering Privacy Contracting
TrustArc Webinar: Mastering Privacy Contracting
TrustArc
 
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
Lorenzo Miniero
 
Contributing to WordPress With & Without Code.pptx
Contributing to WordPress With & Without Code.pptxContributing to WordPress With & Without Code.pptx
Contributing to WordPress With & Without Code.pptx
Patrick Lumumba
 
Dr Jimmy Schwarzkopf presentation on the SUMMIT 2025 A
Dr Jimmy Schwarzkopf presentation on the SUMMIT 2025 ADr Jimmy Schwarzkopf presentation on the SUMMIT 2025 A
Dr Jimmy Schwarzkopf presentation on the SUMMIT 2025 A
Dr. Jimmy Schwarzkopf
 
Agentic AI - The New Era of Intelligence
Agentic AI - The New Era of IntelligenceAgentic AI - The New Era of Intelligence
Agentic AI - The New Era of Intelligence
Muzammil Shah
 
Maxx nft market place new generation nft marketing place
Maxx nft market place new generation nft marketing placeMaxx nft market place new generation nft marketing place
Maxx nft market place new generation nft marketing place
usersalmanrazdelhi
 
Cyber security cyber security cyber security cyber security cyber security cy...
Cyber security cyber security cyber security cyber security cyber security cy...Cyber security cyber security cyber security cyber security cyber security cy...
Cyber security cyber security cyber security cyber security cyber security cy...
pranavbodhak
 
Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025
Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025
Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025
Nikki Chapple
 
Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...
Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...
Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...
Aaryan Kansari
 
STKI Israel Market Study 2025 final v1 version
STKI Israel Market Study 2025 final v1 versionSTKI Israel Market Study 2025 final v1 version
STKI Israel Market Study 2025 final v1 version
Dr. Jimmy Schwarzkopf
 
Gihbli AI and Geo sitution |use/misuse of Ai Technology
Gihbli AI and Geo sitution |use/misuse of Ai TechnologyGihbli AI and Geo sitution |use/misuse of Ai Technology
Gihbli AI and Geo sitution |use/misuse of Ai Technology
zainkhurram1111
 
Offshore IT Support: Balancing In-House and Offshore Help Desk Technicians
Offshore IT Support: Balancing In-House and Offshore Help Desk TechniciansOffshore IT Support: Balancing In-House and Offshore Help Desk Technicians
Offshore IT Support: Balancing In-House and Offshore Help Desk Technicians
john823664
 
Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 ProfessioMaster tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
Kari Kakkonen
 
UiPath Community Zurich: Release Management and Build Pipelines
UiPath Community Zurich: Release Management and Build PipelinesUiPath Community Zurich: Release Management and Build Pipelines
UiPath Community Zurich: Release Management and Build Pipelines
UiPathCommunity
 
Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)
Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)
Marko.js - Unsung Hero of Scalable Web Frameworks (DevDays 2025)
Eugene Fidelin
 
A Comprehensive Guide on Integrating Monoova Payment Gateway
A Comprehensive Guide on Integrating Monoova Payment GatewayA Comprehensive Guide on Integrating Monoova Payment Gateway
A Comprehensive Guide on Integrating Monoova Payment Gateway
danielle hunter
 
New Ways to Reduce Database Costs with ScyllaDB
New Ways to Reduce Database Costs with ScyllaDBNew Ways to Reduce Database Costs with ScyllaDB
New Ways to Reduce Database Costs with ScyllaDB
ScyllaDB
 
Fully Open-Source Private Clouds: Freedom, Security, and Control
Fully Open-Source Private Clouds: Freedom, Security, and ControlFully Open-Source Private Clouds: Freedom, Security, and Control
Fully Open-Source Private Clouds: Freedom, Security, and Control
ShapeBlue
 
Content and eLearning Standards: Finding the Best Fit for Your-Training
Content and eLearning Standards: Finding the Best Fit for Your-TrainingContent and eLearning Standards: Finding the Best Fit for Your-Training
Content and eLearning Standards: Finding the Best Fit for Your-Training
Rustici Software
 
Cyber Security Legal Framework in Nepal.pptx
Cyber Security Legal Framework in Nepal.pptxCyber Security Legal Framework in Nepal.pptx
Cyber Security Legal Framework in Nepal.pptx
Ghimire B.R.
 
TrustArc Webinar: Mastering Privacy Contracting
TrustArc Webinar: Mastering Privacy ContractingTrustArc Webinar: Mastering Privacy Contracting
TrustArc Webinar: Mastering Privacy Contracting
TrustArc
 
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
Multistream in SIP and NoSIP @ OpenSIPS Summit 2025
Lorenzo Miniero
 
Contributing to WordPress With & Without Code.pptx
Contributing to WordPress With & Without Code.pptxContributing to WordPress With & Without Code.pptx
Contributing to WordPress With & Without Code.pptx
Patrick Lumumba
 
Dr Jimmy Schwarzkopf presentation on the SUMMIT 2025 A
Dr Jimmy Schwarzkopf presentation on the SUMMIT 2025 ADr Jimmy Schwarzkopf presentation on the SUMMIT 2025 A
Dr Jimmy Schwarzkopf presentation on the SUMMIT 2025 A
Dr. Jimmy Schwarzkopf
 
Agentic AI - The New Era of Intelligence
Agentic AI - The New Era of IntelligenceAgentic AI - The New Era of Intelligence
Agentic AI - The New Era of Intelligence
Muzammil Shah
 
Maxx nft market place new generation nft marketing place
Maxx nft market place new generation nft marketing placeMaxx nft market place new generation nft marketing place
Maxx nft market place new generation nft marketing place
usersalmanrazdelhi
 
Cyber security cyber security cyber security cyber security cyber security cy...
Cyber security cyber security cyber security cyber security cyber security cy...Cyber security cyber security cyber security cyber security cyber security cy...
Cyber security cyber security cyber security cyber security cyber security cy...
pranavbodhak
 
Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025
Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025
Protecting Your Sensitive Data with Microsoft Purview - IRMS 2025
Nikki Chapple
 
Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...
Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...
Agentic AI Explained: The Next Frontier of Autonomous Intelligence & Generati...
Aaryan Kansari
 
STKI Israel Market Study 2025 final v1 version
STKI Israel Market Study 2025 final v1 versionSTKI Israel Market Study 2025 final v1 version
STKI Israel Market Study 2025 final v1 version
Dr. Jimmy Schwarzkopf
 

Kernel Debugging & Profiling

  • 1. © 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Kernel Debugging & Profiling
  • 2. 2© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. What to Expect? Kernel Debugging Tools & Techniques Kernel Profiling Tools & Mechanism Kernel Testing Ways
  • 3. 3© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Debugging by Printing printk & dmesg int klogctl(int cmd, char *bufp, int len); syslogd #include <syslog.h> void openlog(char *ident, int option, int facility); void syslog ( int priority, char *format, …); void closelog( void ); ident – typically short program name option - LOG_CONS, LOG_PERROR, LOG_PID, … facility - LOG_AUTHPRIV, LOG_CRON, LOG_DAEMON, LOG_USER, ... priority - LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR, LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG klogd – bridge between kernel ring buffer and syslogd
  • 4. 4© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Debugging by Querying Through Kernel Windows /proc Process related stuff General system information: cpuinfo, meminfo, … Drivers related info: devices, iomem, interrupts, ... /sys (Utilities: sysfsutils, sysdiag) Buses, Classes, Devices, ... Using ioctl Through specific drivers Using custom commands
  • 5. 5© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Crash Dumping & Analysis Oops Analysis CONFIG_BUG, CONFIG_KALLSYMS Call Trace cat /proc/kallsyms objdump -d | -S <obj | exe> Crash Dumping (using 2 kernel images – could be same) System Kernel: CONFIG_KEXEC, CONFIG_DEBUG_INFO Dump-capture Kernel: CONFIG_CRASH_DUMP, CONFIG_PROC_VMCORE, Other Architecture specific CONFIG_s Boot System Kernel with crashkernel=size@offset, or crashkernel=range:size Overlay Dump-capture Kernel using kexec After Crash (Simulated using SysRq+c, Or real) System reboots w/ Dump-capture Kernel Collect the dump using kdump, Or simply cp /proc/vmcore <dump file> Analyse using <gdb | crash> vmlinux <dump file>
  • 6. 6© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Kernel Hacking Related Options CONFIG_DEBUG_INFO (gcc -g) CONFIG_DEBUG_FS CONFIG_DYNAMIC_DEBUG CONFIG_DEBUG_STACKOVERFLOW CONFIG_LOCKUP_DETECTOR CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_MUTEXES, ... CONFIG_MAGIC_SYSRQ CONFIG_EARLY_PRINTK CONFIG_DEBUG_SLAB, CONFIG_DEBUG_VM, ... CONFIG_DEBUG_STACK_USAGE CONFIG_PANIC_ON_OOPS CONFIG_PANIC_ON_OOPS_VALUE
  • 7. 7© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Kernel Debuggers Kernel Debugger (Frontend): kdb CONFIG_DEBUG_INFO, CONFIG_MAGIC_SYSRQ CONFIG_KGDB, CONFIG_KGDB_KDB, CONFIG_KGDB_SERIAL_CONSOLE Basic Operations over Serial Console Remote Debugging using kgdb (Kernel Debug Server) Remote connection & debug through gdb vmlinux gdb <kernel src>/vmlinux /proc/kcore Only to gather current kernel debug information
  • 8. 8© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Miscellaneous Tools & Techniques Early Init Debugs CONFIG_SERIAL_EARLYCON CONFIG_EARLY_PRINTK (direct on h/w) Light & Sound JTAG Hardware Protocol Debug Hardware Protocol Analyzers Network Debugging Driver level: ethtool [options] <dev i/f> Packet level: tcpdump [options, e.g. -vv] -i <dev i/f>
  • 9. 9© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Kernel Probes kprobes → CONFIG_KPROBES jprobes → Specialized Kprobes For probing function entry points kretprobes → Return Kprobes For probing function exit points
  • 10. 10© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Tracing Single Process Tracing using strace Traces the system calls made by an application Command: strace [options] <application> Excellent options to tune to get only desired output Linux Trace Toolkit Patch @ http://www.opersys.com/LTT Core Module Code using Trace Services Daemon: tracedaemon Utilities: tracereader, tracevisualizer Function Tracer CONFIG_FUNCTION_TRACER & co /sys/kernel/debug/tracing
  • 11. 11© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Profiling & Code Coverage System Profiling From /proc using LTTng Kernel Profiling: Oprofile CONFIG_PROFILING, CONFIG_OPROFILE, CONFIG_APIC Daemon: oprofiled Utilities: opcontrol, opreport, op_help, ... Kernel Performance Profiling using perf CONFIG_PERF_EVENTS & co stat, list, top, record, report, ... Kernel Code Coverage: CONFIG_GCOV_KERNEL, CONFIG_GCOV_PROFILE_ALL
  • 12. 12© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Testing Possibilities Linux Test Project (LTP) Hosted @ http://ltp.sourceforge.net Suite of around 3000 tests Exercises various parts of the Kernel Mostly automated, except net & storage ones Command: runltp -p -l logfile User Mode Linux (UML) Hosted @ http://user-mode-linux.sourceforge.net A Kernel Instance as a User Mode Process Kernel Debugging without “oops”ing Good for testing & experimenting with unstable kernels
  • 13. 13© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. What all have we learnt? Kernel Debugging Tools & Techniques Debugging by Printing & Querying Crash Dumping & Analysis Kernel Hacking Options Kernel Debuggers Miscellaneous Tools & Techniques Kernel Probes Tracing Kernel Profiling Tools & Mechanism Kernel Testing Ways
  • 14. 14© 2010-16 SysPlay Workshops <[email protected]> All Rights Reserved. Any Queries?