Troubleshooting GitLab CI¶

Overview¶

Diagnose failed jobs, runner problems, auth errors, cache misses, and slow pipelines with a fixed order: lint → config → runner → credentials → cache → performance.

Most “CI is broken” tickets are YAML rules, missing tags, expired tokens, or poisoned caches — not mysterious GitLab bugs. Separate definition failures from execution failures before changing production variables.

This is a core tutorial in Module 17 · Troubleshooting of the REBASH Academy GitLab CI/CD for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites¶

Pipeline Monitoring and Observability
Runner and variables modules completed (or equivalent)

Learning Objectives¶

By the end of this tutorial, you will be able to:

Classify config vs runner vs auth vs cache failures
Use lint, job logs, and CI_* context systematically
Recover from stuck/pending jobs and executor errors
Apply a performance triage for slow pipelines

Architecture¶

This topic’s control points and relationships are shown below.

Theory¶

What it is¶

Troubleshooting GitLab CI is locating which layer failed: YAML/workflow, runner availability, executor runtime, secrets/OIDC, artefacts/cache, or external systems (registry, cloud APIs). A failed job log is necessary but not sufficient — pending jobs never produce a script log.

Symptom	First checks
Pipeline not created	`workflow:rules`, `.gitlab-ci.yml` syntax, CI enabled
Job pending forever	Runner online, tags, protected branches, concurrency
Script exit ≠ 0	Last error in log, image, deps
401 / forbidden	Token, OIDC, protected var, registry login
Flaky / slow	Cache keys, image pulls, serial stages

Why it matters¶

Mean time to recovery for delivery depends on CI as much as production apps. Platform on-call needs a playbook juniors can follow under pressure. The same checks belong in shift-left tooling: glab ci lint and local runners catch definition errors before they burn SaaS minutes.

How it works¶

Use this order every time:

Lint / parse — glab ci lint or YAML load; confirm the job exists for this ref (rules).
Scope — Is the job pending or failed? Pending → runners/tags/protection. Failed → log.
Log — Read from the bottom; note image, before_script, and the failing command.
Auth — Masked variables present on protected branches? OIDC audience/role correct? CI_JOB_TOKEN permissions for cross-project?
Cache / artefacts — Wrong cache:key → cold builds; missing dependencies/needs → missing files.
Runner — gitlab-runner status, executor errors, disk full, Docker privilege, Kubernetes pod events.
Performance — largest jobs by duration; parallelise; slim images; avoid unnecessary needs chains.

Reproduce with a minimal job when possible. Prefer fixing root cause over retry: 2 as a product strategy.

Key concepts and comparisons¶

Failure class	Looks like	Not fixed by
Definition	Job absent / wrong `rules`	Restarting runners
Capacity	Pending, “no runners”	Editing script only
Runtime	Red job, script error	Adding more runners alone
Auth	401/403 mid-job	Clearing cache
Cache	Intermittent missing modules	Random `retry`

Common pitfalls¶

Blaming GitLab.com when the job never matched a runner tag.
Clearing all caches as step one — hides the bad key design.
Putting secrets in logs “just to debug” on shared runners.
Using allow_failure: true to silence a broken gate permanently.

Hands-on Lab¶

Create a workspace for this tutorial.

mkdir -p ~/rebash-gitlab/module-17 && cd ~/rebash-gitlab/module-17

Focus: reproduce a failing job locally and capture triage checklist

Step 1 – Broken job + triage notes¶

cat > .gitlab-ci.yml << 'EOF'
stages: [broken, fixed]
broken_example:
  stage: broken
  image: alpine:3.20
  script: ["curl --version"]
  allow_failure: true
fixed_example:
  stage: fixed
  image: curlimages/curl:8.10.1
  script: ["curl --version"]
EOF
cat > triage.md << 'EOF'
1. First error in job log
2. Confirm image/tag and entrypoint
3. Check rules/needs
4. Verify variables on unprotected branches
5. CI_DEBUG_TRACE only in secure sandbox
EOF

Step 2 – Simulate fix with Docker¶

docker run --rm alpine:3.20 sh -c 'command -v curl || echo curl-missing'
docker run --rm curlimages/curl:8.10.1 curl --version | head -n 1
test -f triage.md

Final step – Cleanup note¶

docker rmi alpine:3.20 curlimages/curl:8.10.1 2>/dev/null || true
# Keep ~/rebash-gitlab/ for later tutorials

Validation¶

Lab commands run under ~/rebash-gitlab/module-17/
You can explain each Theory section in your own words
You used modern tooling where it applies to this topic
You can describe one production failure mode for this topic

Code Walkthrough¶

Production practice for Troubleshooting GitLab CI always combines:

Inspect before you change (status, plan, logs, dry-run)
Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
Capture evidence (command output, pipeline logs) for handovers
Prefer current tools and APIs over legacy shortcuts
Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations¶

Treat credentials and tokens for gitlab as privileged — never commit them
Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
Validate blast radius before apply/deploy/delete operations
Restrict who can approve production changes
Collect audit logs; limit who can read sensitive traces

Common Mistakes¶

Blaming GitLab.com when the job never matched a runner tag.

Validate assumptions against the Theory section and official docs before changing production.

Clearing all caches as step one — hides the bad key design.

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices¶

Encode Troubleshooting GitLab CI changes as code and review them in pull requests
Pin versions (images, modules, actions, provider plugins)
Separate environments with clear promotion gates
Alert on symptoms with runbooks attached
Destroy lab resources; tag everything with owner and expiry where possible