Troubleshooting Docker Containers¶

Overview¶

Apply a systematic playbook: status → logs → inspect → exec → host resources — for the common Docker failure modes.

Most incidents are config or resource issues, not “Docker is broken.” Reproduce locally with the same image digest when possible.

This is a core tutorial in Module 16 · Troubleshooting of the REBASH Academy Docker for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites¶

Docker in CI/CD Pipelines

Learning Objectives¶

By the end of this tutorial, you will be able to:

Diagnose exit codes and restart loops
Fix image pull / auth errors
Debug port and DNS issues
Spot permission / volume UID problems
Free disk with informed prune

Architecture¶

This topic’s control points and relationships are shown below.

Theory¶

What¶

Container troubleshooting follows a log-first, state-second method: read exit codes and logs, inspect configuration, verify networks and mounts, then check host resources. Symptoms include crash loops, pull failures, connection refusals, and permission errors on volumes.

Why¶

Guessing wastes time. The same patterns appear from laptop Docker to Kubernetes CrashLoopBackOff — the process failed, the image failed to pull, or the environment is wrong. A repeatable checklist makes on-call work calmer.

How it works¶

If a container will not stay up, docker logs and docker inspect (ExitCode, Error, OOMKilled) come first. Instant exit often means the main process crashed — override the command with a sleep/shell to explore the filesystem when appropriate. Pull errors point at auth, tag typos, or rate limits. Connection issues need published ports, network membership, and host firewall checks. Permission denied on mounted files usually means UID mismatch with USER. Disk-full failures show in daemon errors; docker system df confirms.

Symptom	Checks
Won't start	Logs, ExitCode, CMD
Instant exit	App crash — override command
Pull errors	Auth, tag, rate limit
Can't connect	`-p`, network, firewall
Permission denied	Volume UID vs `USER`
No space	`docker system df`, prune

Key concepts¶

Exit code literacy — 137 often means SIGKILL/OOM
Layer the problem — app vs image vs runtime vs host
Reproduce minimally — smallest docker run that fails
Time correlation — deploys, pulls, and node pressure

Common pitfalls¶

Deleting the failed container before reading logs
Fixing the host when the CMD is wrong
Ignoring registry rate limits and “random” pull failures
Restarting endlessly without a health/backoff strategy

Hands-on Lab¶

Create a workspace for this tutorial.

mkdir -p ~/rebash-docker/module-16 && cd ~/rebash-docker/module-16

Focus: debug a failing container with logs, inspect, and a corrected run

Step 1 – Broken then fixed¶

docker run -d --name rebash-bad alpine:3.20 sh -c 'echo boom; exit 1'
sleep 1
docker ps -a --filter name=rebash-bad
docker logs rebash-bad
docker inspect rebash-bad --format 'exit={{.State.ExitCode}}'
docker rm -f rebash-bad
docker run -d --name rebash-good alpine:3.20 sh -c 'echo ok; sleep 30'
docker logs rebash-good

Step 2 – Cleanup¶

docker rm -f rebash-bad rebash-good

Final step – Cleanup note¶

docker rm -f rebash-bad rebash-good 2>/dev/null || true
# Keep ~/rebash-docker/ for later tutorials

Validation¶

Lab commands run under ~/rebash-docker/module-16/
You can explain each Theory section in your own words
You used modern tooling where it applies to this topic
You can describe one production failure mode for this topic

Code Walkthrough¶

Production practice for Troubleshooting Docker Containers always combines:

Inspect before you change (status, plan, logs, dry-run)
Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
Capture evidence (command output, pipeline logs) for handovers
Prefer current tools and APIs over legacy shortcuts
Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations¶

Treat credentials and tokens for docker as privileged — never commit them
Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
Validate blast radius before apply/deploy/delete operations
Restrict who can approve production changes
Collect audit logs; limit who can read sensitive traces

Common Mistakes¶

Deleting the failed container before reading logs

Validate assumptions against the Theory section and official docs before changing production.

Fixing the host when the CMD is wrong

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices¶

Encode Troubleshooting Docker Containers changes as code and review them in pull requests
Pin versions (images, modules, actions, provider plugins)
Separate environments with clear promotion gates
Alert on symptoms with runbooks attached
Destroy lab resources; tag everything with owner and expiry where possible