Linux Kernel Crash Debugging

This skill guides you through analyzing Linux kernel crash dumps using the crash utility.

Installation

Claude Code

CODEBLOCK0

OpenClaw

CODEBLOCK1

Quick Start

Starting a Session

CODEBLOCK2

Core Debugging Workflow

CODEBLOCK3

🤖 Agent Execution Directives

If you are an AI/Agent using this skill, do not invoke crash interactively as it will block your subshell.

1. 1. Use the bundled wrapper ./scripts/agent-crash.sh which maps precisely to the workflows below but safely truncates outputs:

- ./scripts/agent-crash.sh -k vmlinux -c vmcore triage - Safely runs initial sys, log, and bt. - ./scripts/agent-crash.sh -k vmlinux -c vmcore flow-oom - Top 15 memory checks. - ./scripts/agent-crash.sh -k vmlinux -c vmcore flow-deadlock - Pulls UN task stacks. - ./scripts/agent-crash.sh -k vmlinux -c vmcore dis-regs <func> <pid> - Assembly regression. - ./scripts/agent-crash.sh -k vmlinux -c vmcore check-poison <addr> - Pattern match memory poisons.

1. 2. Fallback Strategy: If macros don't solve the issue, fall back to basic primitives manually: ./scripts/agent-crash.sh -k vmlinux -c vmcore run "rd ffff880123456780".
2. Check references/agentic-heuristics.md for extended expert methodologies.

Prerequisites

Item	Requirement
vmlinux	Must have debug symbols (CONFIG_DEBUG_INFO=y)
vmcore

kdump/netdump/diskdump/ELF format | | Version | vmlinux must exactly match the vmcore kernel version |

Package Installation

Anolis OS / Alibaba Cloud Linux

CODEBLOCK4

RHEL / CentOS / Rocky / AlmaLinux

CODEBLOCK5

Ubuntu / Debian

CODEBLOCK6

Self-compiled Kernel

CODEBLOCK7

Verify Installation

CODEBLOCK8

Core Command Reference

Debugging Analysis

Command	Purpose	Example
INLINECODE13	System info/panic reason	INLINECODE14, INLINECODE15
INLINECODE16

Kernel message buffer | log, log \| tail | | bt | Stack backtrace | bt, bt -a, bt -f | | struct | View structures | struct task_struct <addr> | | p/px/pd | Print variables | p jiffies, px current | | kmem | Memory analysis | kmem -i, kmem -S <cache> |

Tasks and Processes

Command	Purpose	Example
INLINECODE31	Process list	INLINECODE32, INLINECODE33
INLINECODE34

Switch context | set <pid>, set -p | | foreach | Batch task operations | foreach bt, foreach UN bt | | task | task_struct contents | task <pid> | | files | Open files | files <pid> |

Memory Operations

Command	Purpose	Example
INLINECODE44	Read memory	INLINECODE45, INLINECODE46
INLINECODE47

Search memory | search -k deadbeef | | vtop | Address translation | vtop <addr> | | list | Traverse linked lists | list task_struct.tasks -h <addr> |

bt Command Details

The most important debugging command:

CODEBLOCK9

Context Management

Crash session has a "current context" affecting bt, files, vm commands:

CODEBLOCK10

Session Control

CODEBLOCK11

Typical Debugging Scenarios

Kernel BUG Location

CODEBLOCK12

Deadlock Analysis

CODEBLOCK13

Memory Issues

CODEBLOCK14

Stack Overflow

CODEBLOCK15

Advanced Techniques

Chained Queries

CODEBLOCK16

Batch Slab Inspection

CODEBLOCK17

Kernel Linked List Traversal

CODEBLOCK18

Extended Reference

For detailed information, refer to the following reference files:

File	Content
INLINECODE56	Advanced commands: list, rd, search, vtop, kmem, foreach
INLINECODE57

vmcore file format, ELF structure, VMCOREINFO |
| references/case-studies.md | Debugging cases: kernel BUG, deadlock, OOM, NULL pointer, stack overflow |
| references/debug-tools-guide.md | Advanced debugging tools: KASAN, Kprobes, Kmemleak, UBSAN (require kernel rebuild) |

Usage:
CODEBLOCK19

Common Errors

CODEBLOCK20

Security Warnings

⚠️ Dangerous Operations

The following commands can cause system damage or data loss:

Command	Risk	Recommendation
INLINECODE60	Writes to live kernel memory	NEVER use on production systems - can crash or corrupt running kernel
GDB passthrough

Unrestricted memory access | Use with caution, may modify memory or registers |

🔒 Sensitive Data Handling

• - vmcore files contain complete kernel memory, potentially including:

- User process memory and credentials - Encryption keys and secrets - Network connection data and passwords

• - Access control: Restrict vmcore file access to authorized personnel
• Secure storage: Store dump files in encrypted or access-controlled directories
• Secure disposal: Use shred or secure delete when disposing of vmcore files

🛡️ Best Practices

1. 1. Only analyze vmcore files in isolated/test environments when possible
2. Never share raw vmcore files publicly without sanitization
3. Consider using makedumpfile -d to filter sensitive pages before analysis
4. Document and audit all crash analysis sessions for compliance

Important Notes

1. 1. Version Match: vmlinux must exactly match the vmcore kernel version
2. Debug Info: Must use vmlinux with debug symbols
3. Context Awareness: bt, files, vm commands are affected by current context
4. Live System Modification: wr command modifies running kernel, extremely dangerous

Resources

• - Crash Utility Whitepaper
• Crash Utility Documentation
• Crash Help Pages

Contributing

This is an open-source project. Contributions are welcome!

• - GitHub Repository: https://github.com/crazyss/linux-kernel-crash-debug
• Report Issues: GitHub Issues
• Submit PRs: Pull requests are welcome for bug fixes, new features, or documentation improvements

See CONTRIBUTING.md for guidelines.