Kubernetes Debugging Skill

Overview

Systematic toolkit for debugging Kubernetes clusters, workloads, networking, and storage with a deterministic, safety-first workflow.

Trigger Phrases

Use this skill when requests resemble:

• - "My pod is in CrashLoopBackOff; help me find the root cause."
• "Service DNS works in one pod but not another."
• "Deployment rollout is stuck."
• "Pods are Pending and not scheduling."
• "Cluster health looks degraded after a change."
• "PVC is pending and pods cannot mount storage."

Prerequisites

Run from the skill directory (devops-skills-plugin/skills/k8s-debug) so relative script paths work as written.

Required

• - kubectl installed and configured.
• An active cluster context.
• Read access to namespaces, pods, events, services, and nodes.

Quick preflight:

CODEBLOCK0

Optional but Recommended

• - jq for more precise filtering in ./scripts/cluster_health.sh.
• Metrics API (metrics-server) for kubectl top.
• In-container debug tools (nslookup, getent, curl, wget, ip) for deep network tests.

Fallback behavior:

• - If optional tools are missing, scripts continue and print warnings with reduced output.
• If kubectl top is unavailable, continue with kubectl describe and events.

When to Use This Skill

Use this skill for:

• - Pod failures (CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled)
• Service connectivity or DNS resolution issues
• Network policy or ingress problems
• Volume and storage mount failures
• Deployment rollout issues
• Cluster health or performance degradation
• Resource exhaustion (CPU/memory)
• Configuration problems (ConfigMaps, Secrets, RBAC)

Safety Rules for Disruptive Commands

Default mode is read-only diagnosis first. Only execute disruptive commands after confirming blast radius and rollback.

Commands requiring explicit confirmation:

• - INLINECODE15
• INLINECODE16
• INLINECODE17
• INLINECODE18
• INLINECODE19

Before disruptive actions:
CODEBLOCK1

Reference Navigation Map

Load only the section needed for the observed symptom.

Symptom / Need	Open	Start section
You need an end-to-end diagnosis path	INLINECODE20	INLINECODE21
Pod state is Pending, CrashLoopBackOff, or INLINECODE24

./references/troubleshooting_workflow.md | Pod Lifecycle Troubleshooting |
| Service reachability or DNS failure | ./references/troubleshooting_workflow.md | Network Troubleshooting Workflow |
| Node pressure or performance regression | ./references/troubleshooting_workflow.md | Resource and Performance Workflow |
| PVC / PV / storage class issues | ./references/troubleshooting_workflow.md | Storage Troubleshooting Workflow |
| Quick symptom-to-fix lookup | ./references/common_issues.md | matching issue heading |
| Post-mortem fix options for known issues | ./references/common_issues.md | Solutions sections |

Scripts Overview

Script	Purpose	Required args	Optional args	Output	Fallback behavior
INLINECODE36	Cluster-wide health snapshot (nodes, workloads, events, common failure states)	None	INLINECODE37, K8S_REQUEST_TIMEOUT env var	Sectioned report to stdout	Continues on check failures, tracks them in summary and exit code
INLINECODE39

Pod-centric network and DNS diagnostics | <pod-name> (<namespace> defaults to default) | --strict, --insecure, K8S_REQUEST_TIMEOUT env var | Sectioned report to stdout | Uses secure API probe by default; insecure TLS requires explicit --insecure | | ./scripts/pod_diagnostics.py | Deep pod diagnostics (status, describe, YAML, events, per-container logs, node context) | <pod-name> | -n/--namespace, -o/--output | Sectioned report to stdout or file | Fails fast on missing access; skips optional metrics/log blocks with clear messages |

Script Exit Codes

INLINECODE51 and ./scripts/network_debug.sh share the same contract:

• - 0: checks completed with no check failures (warnings allowed unless --strict is set).
• INLINECODE55: one or more checks failed, or warnings occurred in --strict mode.
• INLINECODE57: blocked preconditions (for example: missing kubectl, no active context, inaccessible namespace/pod).

Deterministic Debugging Workflow

Follow this systematic approach for any Kubernetes issue:

1. Preflight and Scope

CODEBLOCK2

If preflight fails, stop and fix access/context first.

2. Identify the Problem Layer

Categorize the issue:

• - Application Layer: Application crashes, errors, bugs
• Pod Layer: Pod not starting, restarting, or pending
• Service Layer: Network connectivity, DNS issues
• Node Layer: Node not ready, resource exhaustion
• Cluster Layer: Control plane issues, API problems
• Storage Layer: Volume mount failures, PVC issues
• Configuration Layer: ConfigMap, Secret, RBAC issues

3. Gather Diagnostics with the Right Script

Use the appropriate diagnostic script based on scope:

Pod-Level Diagnostics

Use ./scripts/pod_diagnostics.py for comprehensive pod analysis:

CODEBLOCK3

This script gathers:

• - Pod status and description
• Pod events
• Container logs (current and previous)
• Resource usage
• Node information
• YAML configuration

Output can be saved for analysis:

CODEBLOCK4

Cluster-Level Health Check

Use ./scripts/cluster_health.sh for overall cluster diagnostics:

CODEBLOCK5

This script checks:

• - Cluster info and version
• Node status and resources
• Pods across all namespaces
• Failed/pending pods
• Recent events
• Deployments, services, statefulsets, daemonsets
• PVCs and PVs
• Component health
• Common error states (CrashLoopBackOff, ImagePullBackOff)

Network Diagnostics

Use ./scripts/network_debug.sh for connectivity issues:

CODEBLOCK6

This script analyzes:

• - Pod network configuration
• DNS setup and resolution
• Service endpoints
• Network policies
• Connectivity tests
• CoreDNS logs

4. Follow Issue-Specific Reference Workflow

Based on the identified issue, consult ./references/troubleshooting_workflow.md:

• - Pod Pending: Resource/scheduling workflow
• CrashLoopBackOff: Application crash workflow
• ImagePullBackOff: Image pull workflow
• Service issues: Network connectivity workflow
• DNS failures: DNS troubleshooting workflow
• Resource exhaustion: Performance investigation workflow
• Storage issues: PVC binding workflow
• Deployment stuck: Rollout workflow

5. Apply Targeted Fixes

Refer to ./references/common_issues.md for symptom-specific fixes.

6. Verify and Close

Run final verification:

CODEBLOCK7

Issue is done when user-visible behavior is healthy and no new critical warning events appear.

Example Flows

Example 1: CrashLoopBackOff in payments Namespace

CODEBLOCK8

Then open ./references/common_issues.md and apply the CrashLoopBackOff solutions.

Example 2: Service DNS/Connectivity Failure

CODEBLOCK9

Then follow Service Connectivity Workflow in ./references/troubleshooting_workflow.md.

Essential Manual Commands

Pod Debugging

CODEBLOCK10

Service and Network Debugging

CODEBLOCK11

Resource Monitoring

CODEBLOCK12

Emergency Operations

CODEBLOCK13

Completion Criteria

Troubleshooting session is complete when all are true:

• - [ ] Cluster context and namespace are confirmed.
• [ ] Relevant diagnostic script output is captured.
• [ ] Root cause is identified and tied to evidence (events/logs/config/state).
• [ ] Any disruptive action was preceded by snapshot and rollback plan.
• [ ] Fix verification commands show healthy state.
• [ ] Reference path used (./references/troubleshooting_workflow.md or ./references/common_issues.md) is documented in notes.

Related Tools

Useful additional tools for Kubernetes debugging:

• - kubectl-debug: Advanced debugging plugin
• stern: Multi-pod log tailing
• kubectx/kubens: Context and namespace switching
• k9s: Terminal UI for Kubernetes
• lens: Desktop IDE for Kubernetes
• Prometheus/Grafana: Monitoring and alerting
• Jaeger/Zipkin: Distributed tracing