Lab — Docker Compose Stack Recovery

Lab Overview

Purpose: Practise recovering a multi-service Compose stack the way platform engineers debug staging outages.

Scenario: web and api are defined in Compose, but the stack is unhealthy. Customers see connection errors. You must find root cause with compose ps, logs, and networking checks — then fix and prove health.

Expected outcome: Both services are healthy/running, curl succeeds to the published web port, and you can state the root cause in one sentence.

This is a lab, not a tutorial

Apply Docker track skills under incomplete information. Prefer evidence over guessing.

Business Scenario

A product team ships an internal status page (web) that calls a small API (api) over a Compose bridge network. A Friday deploy broke staging: containers restart or fail health checks. There is no Kubernetes yet for this app — Compose is the runtime. You own recovery before the Monday demo.

Learning Objectives

By the end of this lab, you will be able to:

• Inspect Compose service state, ports, and health
• Correlate container logs with misconfigured ports, environment, and healthcheck
• Repair compose.yaml and rebuild/recreate services safely
• Validate end-to-end HTTP behaviour
• Tear down volumes and networks cleanly

Prerequisites

Knowledge

• Introduction to Containers and Docker
• Docker Compose Fundamentals
• Docker Networking Fundamentals
• Troubleshooting Docker Containers
• Environment Variables and Secrets

Software

Tool	Notes
Docker Engine + Compose v2	docker compose version
curl	health checks
2–4 GB free disk	image pulls

Estimated cost: £0 locally.

Architecture

Environment

Laptop or Linux VM with Docker. Do not run against a shared production host.

Initial State

You will create a broken Compose project on purpose, then triage it.

Lab Tasks

Task 1 — Create the broken lab project

Objective: Materialise the failing staging stack.

mkdir -p ~/rebash-lab-compose/{api,web} && cd ~/rebash-lab-compose

cat > api/Dockerfile <<'EOF'
FROM python:3.12-alpine
WORKDIR /app
COPY server.py .
EXPOSE 8080
CMD ["python", "server.py"]
EOF

cat > api/server.py <<'EOF'
from http.server import BaseHTTPRequestHandler, HTTPServer
import os

TOKEN = os.environ.get("API_TOKEN", "")

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/healthz":
            body = b'{"status":"ok","service":"api"}\n'
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return
        if self.path == "/status":
            if TOKEN and self.headers.get("X-API-Token") != TOKEN:
                self.send_error(401)
                return
            body = b'{"app":"rebash-status","env":"lab"}\n'
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return
        self.send_error(404)
    def log_message(self, *args):
        return

HTTPServer(("0.0.0.0", 8080), H).serve_forever()
EOF

cat > web/Dockerfile <<'EOF'
FROM python:3.12-alpine
WORKDIR /app
COPY proxy.py .
EXPOSE 8000
CMD ["python", "proxy.py"]
EOF

cat > web/proxy.py <<'EOF'
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.request import Request, urlopen
import os

API = os.environ.get("API_URL", "http://api:8080")
TOKEN = os.environ.get("API_TOKEN", "")

class H(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/healthz":
            body = b'{"status":"ok","service":"web"}\n'
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
            return
        try:
            req = Request(API + "/status", headers={"X-API-Token": TOKEN})
            with urlopen(req, timeout=3) as r:
                body = r.read()
            self.send_response(200)
            self.send_header("Content-Type", "application/json")
            self.send_header("Content-Length", str(len(body)))
            self.end_headers()
            self.wfile.write(body)
        except Exception as e:
            msg = ("upstream error: " + str(e)).encode()
            self.send_response(502)
            self.send_header("Content-Type", "text/plain")
            self.send_header("Content-Length", str(len(msg)))
            self.end_headers()
            self.wfile.write(msg)
    def log_message(self, *args):
        return

HTTPServer(("0.0.0.0", 8000), H).serve_forever()
EOF

# INTENTIONAL BUGS: wrong published port, wrong API_URL host, missing token on web
cat > compose.yaml <<'EOF'
services:
  api:
    build: ./api
    environment:
      API_TOKEN: "lab-secret"
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8080/healthz"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks: [rebash-net]

  web:
    build: ./web
    ports:
      - "18080:8000"
    environment:
      API_URL: "http://api.internal:8080"
      # API_TOKEN intentionally omitted
    depends_on:
      - api
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8000/healthz"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks: [rebash-net]

networks:
  rebash-net:
EOF

Alpine note: wget may be missing — if healthchecks fail for that reason, install it in Dockerfiles (RUN apk add --no-cache wget) as part of your fix, or switch healthchecks to python -c ....

Validation:

test -f compose.yaml && echo "project ready"

Task 2 — Bring the stack up and observe failure

Objective: Capture evidence before editing.

cd ~/rebash-lab-compose
docker compose up -d --build
docker compose ps
docker compose logs --no-color --tail=80
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:18080/ || true
curl -sS http://127.0.0.1:18080/ || true

Expected output: web may be up but / returns 502; or healthchecks fail. Logs show connection or 401 errors.

Validation: Note one symptom and one hypothesis before changing YAML.

Task 3 — Network and env triage

Objective: Confirm DNS/name and token issues.

docker compose exec api wget -qO- http://127.0.0.1:8080/healthz || \
  docker compose exec api python -c "import urllib.request;print(urllib.request.urlopen('http://127.0.0.1:8080/healthz').read().decode())"
docker compose exec web printenv | sort | grep -E 'API_|TOKEN' || true
docker compose exec web getent hosts api || docker compose exec web nslookup api || true

Expected output: Service DNS name is api, not api.internal. Token missing on web.

Task 4 — Repair compose and recreate

Objective: Minimal fix: correct API_URL, pass API_TOKEN, keep port mapping intentional.

cat > compose.yaml <<'EOF'
services:
  api:
    build: ./api
    environment:
      API_TOKEN: "lab-secret"
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/healthz')"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks: [rebash-net]

  web:
    build: ./web
    ports:
      - "18080:8000"
    environment:
      API_URL: "http://api:8080"
      API_TOKEN: "lab-secret"
    depends_on:
      api:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz')"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks: [rebash-net]

networks:
  rebash-net:
EOF

docker compose up -d --build
docker compose ps

Expected output: Both services healthy/running.

Validation:

curl -sS http://127.0.0.1:18080/healthz
curl -sS http://127.0.0.1:18080/

Task 5 — Harden and document

Objective: Remove secrets from plaintext defaults for the write-up (still a lab).

# Demonstrate env file pattern (optional)
printf 'API_TOKEN=lab-secret\n' > .env
grep -n API_TOKEN compose.yaml

In your notes: root cause (wrong hostname + missing token) and the verification curls.

Validation

Check	Pass criteria
Compose ps	api and web running; health OK if used
Web health	GET /healthz → 200
Upstream	GET / → JSON with rebash-status
Root cause	Hostname api.internal + missing token explained

Troubleshooting

Symptoms	Causes	Resolution	Verification
502 from web	Bad API_URL	Use Compose service name api	curl /
401 upstream	Token mismatch	Align API_TOKEN on both services	curl /
Healthcheck fail	Missing wget	Python one-liner healthcheck	compose ps
Port conflict	18080 in use	Change host port mapping	ss / compose ps

Challenge Extensions

• Add a reverse proxy service and TLS locally
• Scan images with Trivy before “merge”
• Split override files for compose.override.yaml

Cleanup

cd ~/rebash-lab-compose
docker compose down -v --remove-orphans
cd ~
rm -rf ~/rebash-lab-compose

Production Discussion

Production Compose (or Swarm/K8s) would use secrets stores, not .env in Git; pin digests; run non-root users; centralise logs; gate deploys on healthchecks in CI.

Best Practices

• Prefer service DNS names on the Compose network
• Healthchecks should exercise the real listener
• Recreate after config changes (up -d --build)
• Never commit real tokens

Common Mistakes

Mistake	Why	Fix
Publishing every container port	Habit	Publish only the edge service
Using localhost inside containers	Host mental model	Use service names
depends_on without health	Race	condition: service_healthy

Success Criteria

You restored a two-service stack from evidence, validated HTTP, and cleaned up.

Reflection Questions

1. Why did api.internal fail while api works?
2. How would you prevent token drift between services?
3. What would change if web ran in Kubernetes instead?

Interview Connection

Expect Compose networking, healthchecks, and secret-handling questions. Use this recovery story. See Docker Interview Prep.

Related Tutorials

• Docker
• Docker Compose Fundamentals
• Troubleshooting Docker Containers
• Cheat sheet: Docker
• Next lab: Kubernetes Deployment Triage
• Project: Status API Portfolio Build

References

1. Docker Compose specification
2. Docker networking
3. Container healthchecks