Troubleshooting GitHub Actions¶

Overview¶

Diagnose failed jobs, runner problems, authentication errors, cache misses, deploy failures, and slow workflows with a fixed order: trigger → permissions → runner → credentials → cache → deploy target → performance.

Most “Actions is broken” tickets are skipped jobs, missing permissions, offline self-hosted runners, expired OIDC trust, or poisoned caches — not mysterious GitHub bugs. Separate definition failures (workflow never ran the job you expected) from execution failures before changing production secrets.

This is a core tutorial in Module 16 · Troubleshooting of the REBASH Academy GitHub Actions for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites¶

Production Pipelines and Environments
Runner, secrets/OIDC, and cache modules completed (or equivalent)

Learning Objectives¶

By the end of this tutorial, you will be able to:

Classify trigger / permissions / runner / auth / cache / deploy failures
Use job logs, CONTEXT dumps, and act-free local YAML checks systematically
Recover from queued jobs and self-hosted executor errors
Apply a performance triage for slow workflows

Architecture¶

This topic’s control points and relationships are shown below.

Theory¶

What it is¶

Troubleshooting GitHub Actions locates which layer failed: trigger/if, permissions, runner capacity, step runtime, secrets/OIDC, cache/artefacts, or the deploy target. A red step log is necessary but not sufficient — queued jobs never produce script output.

Symptom	First checks
Workflow not listed	`on:` / paths / workflow enabled
Job skipped	Job `if`, failed/skipped `needs`
Queued forever	Labels, concurrency, self-hosted online
Step exit ≠ 0	Failing command, tool/image version
401 / forbidden	`permissions`, secrets, OIDC, environments
Flaky / slow	Cache keys, cold images, serial `needs`

Why it matters¶

Delivery recovery depends on CI as much as apps. Platform on-call needs a playbook juniors can follow under pressure. Shift-left YAML checks and action-pin review catch definition errors before they burn hosted minutes.

How it works¶

Trigger — Did on: match?
Graph — Skipped via needs / if?
Permissions — Least privilege (contents, id-token, packages)?
Runner — Labels match? Self-hosted up, disk, Docker?
Log — Read the failing step; use step debug only in safe repos.
Auth — Secrets/OIDC/environment gates/registry login?
Cache — Wrong key, missing needs, or corrupt restore?
Deploy / perf — Target RBAC/probes/approvals; then longest jobs and serial chains.

Prefer root-cause fixes over retry-as-strategy. Reproduce with a minimal workflow when stuck.

Key concepts and comparisons¶

Failure class	Looks like	Not fixed by
Definition	Job absent / skipped	Restarting runners
Capacity	Queued, no runners	Editing script only
Runtime	Red step	Adding runners alone
Auth	401/403	Clearing cache
Cache / deploy	Missing files / target reject	Blind re-run

Common pitfalls¶

Blaming GitHub.com when no runner matches runs-on labels.
Clearing all caches first — hides bad key design.
Echoing secrets into logs on shared runners.
continue-on-error: true as a permanent broken gate.
Re-running while an Environment still awaits approval.

Hands-on Lab¶

Create a workspace for this tutorial.

mkdir -p ~/rebash-github-actions/module-16/.github/workflows && cd ~/rebash-github-actions/module-16/.github/workflows

Focus: capture a failing step pattern and local reproduction

Step 1 – Broken vs fixed workflow pair¶

mkdir -p .github/workflows
cat > triage.md << 'EOF'
1. Open the failed step log — first error
2. Confirm action version
3. Check permissions for GITHUB_TOKEN
4. Re-run failed jobs; debug logging only in private forks
5. Reproduce steps in a clean container
EOF
cat > .github/workflows/troubleshoot.yml << 'EOF'
name: Troubleshoot
on: [workflow_dispatch]
permissions:
  contents: read
jobs:
  broken_shape:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - run: curl --fail https://example.invalid/missing
  fixed_shape:
    runs-on: ubuntu-latest
    steps:
      - run: curl --fail -I https://example.com | head -n 5
EOF

Step 2 – Reproduce with Docker¶

docker run --rm curlimages/curl:8.10.1 curl --fail -I https://example.com | head -n 5
test -f triage.md
docker rmi curlimages/curl:8.10.1 2>/dev/null || true

Final step – Cleanup note¶

docker rmi curlimages/curl:8.10.1 2>/dev/null || true
# Keep ~/rebash-github-actions/ for later tutorials

Validation¶

Lab commands run under ~/rebash-github-actions/module-16/.github/workflows/
You can explain each Theory section in your own words
You used modern tooling where it applies to this topic
You can describe one production failure mode for this topic

Code Walkthrough¶

Production practice for Troubleshooting GitHub Actions always combines:

Inspect before you change (status, plan, logs, dry-run)
Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
Capture evidence (command output, pipeline logs) for handovers
Prefer current tools and APIs over legacy shortcuts
Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations¶

Treat credentials and tokens for github-actions as privileged — never commit them
Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
Validate blast radius before apply/deploy/delete operations
Restrict who can approve production changes
Collect audit logs; limit who can read sensitive traces

Common Mistakes¶

Blaming GitHub.com when no runner matches runs-on labels.

Validate assumptions against the Theory section and official docs before changing production.

Clearing all caches first — hides bad key design.

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices¶

Encode Troubleshooting GitHub Actions changes as code and review them in pull requests
Pin versions (images, modules, actions, provider plugins)
Separate environments with clear promotion gates
Alert on symptoms with runbooks attached
Destroy lab resources; tag everything with owner and expiry where possible