Linux

# Linux Gotchas ## Permission Traps - `chmod 777` fixes nothing, breaks everything — find the actual owner/group issue - Setuid on scripts is ignored for security — only works on binaries - `chown -R` follows symlinks outside target directory — use `--no-dereference` - Default umask 022 makes files world-readable — set 077 for sensitive systems - ACLs override traditional permissions silently — check with `getfacl` ## Process Gotchas - `kill` sends SIGTERM by default, not SIGKILL — process can ignore it - `nohup` doesn't work if process already running — use `disown` instead - Background job with `&` still dies on terminal close without `disown` or `nohup` - Zombie processes can't be killed — parent must call wait() or be killed - `kill -9` skips cleanup handlers — data loss possible, use SIGTERM first ## Filesystem Traps - Deleting open file doesn't free space until process closes it — check `lsof +L1` - `rm -rf /path /` with accidental space = disaster — use `rm -rf /path/` trailing slash - Inodes exhausted while disk shows space free — many small files problem - Symlink loops cause infinite recursion — `find -L` follows them - `/tmp` cleared on reboot — don't store persistent data there ## Disk Space Mysteries - Deleted files held open by processes — `lsof +L1` shows them, restart process to free - Reserved blocks (5% default) only for root — `tune2fs -m 1` to reduce - Journal eating space — `journalctl --vacuum-size=500M` - Docker overlay eating space — `docker system prune -a` - Snapshots consuming space — check LVM, ZFS, or cloud provider snapshots ## Networking - `localhost` and `127.0.0.1` may resolve differently — check `/etc/hosts` - Firewall rules flushed on reboot unless saved — `iptables-save` or use firewalld/ufw persistence - `netstat` deprecated — use `ss` instead - Port below 1024 requires root — use `setcap` for capability instead - TCP TIME_WAIT exhaustion under load — tune `net.ipv4.tcp_tw_reuse` ## SSH Traps - Wrong permissions on ~/.ssh = silent auth failure — 700 for dir, 600 for keys - Agent forwarding exposes your keys to remote admins — avoid on untrusted servers - Known hosts hash doesn't match after server rebuild — remove old entry with `ssh-keygen -R` - SSH config Host blocks: first match wins — put specific hosts before wildcards - Connection timeout on idle — add `ServerAliveInterval 60` to config ## Systemd - `systemctl enable` doesn't start service — also need `start` - `restart` vs `reload`: restart drops connections, reload doesn't (if supported) - Journal logs lost on reboot by default — set `Storage=persistent` in journald.conf - Failed service doesn't retry by default — add `Restart=on-failure` to unit - Dependency on network: `After=network.target` isn't enough — use `network-online.target` ## Cron Pitfalls - Cron has minimal PATH — use absolute paths or set PATH in crontab - Output goes to mail by default — redirect to file or `/dev/null` - Cron uses system timezone, not user's — set TZ in crontab if needed - Crontab lost if edited incorrectly — `crontab -l > backup` before editing - @reboot runs on daemon restart too, not just system reboot ## Memory and OOM - OOM killer picks "best" victim, often not the offender — check dmesg for kills - Swap thrashing worse than OOM — monitor with `vmstat` - Memory usage in `free` includes cache — "available" is what matters - Process memory in `/proc/[pid]/status` — VmRSS is actual usage - cgroups limit respected before system OOM — containers die first ## Commands That Lie - `df` shows filesystem capacity, not physical disk — check underlying device - `du` doesn't count sparse files correctly — file appears smaller than disk usage - `ps aux` memory percentage can exceed 100% (shared memory counted multiple times) - `uptime` load average includes uninterruptible I/O wait — not just CPU - `top` CPU percentage is per-core — 400% means 4 cores maxed