Troubleshooting Python Automation¶

Overview¶

Run a systematic checklist for failing automation: environment and pins, imports/deps, API/auth failures, performance/memory, then fix with a regression test.

“Works on my machine” is an environment bug until proven otherwise. Fingerprint the interpreter, recreate the venv, bisect the last change, and capture evidence (logs, exit codes, minimal repro).

This closes the Python for DevOps Engineers tutorial track. Diagrams use Excalidraw only.

This is a core tutorial in Module 27 · Troubleshooting of the REBASH Academy Python for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites¶

Required¶

Modules 1, 8, 10, 14, and 24 completed (or equivalent experience)

Learning Objectives¶

By the end of this tutorial, you will be able to:

Fingerprint Python/venv/pins in an incident
Resolve common ModuleNotFoundError / dependency conflicts
Triage HTTP/SDK auth and timeout failures
Approach memory growth and slow jobs
Produce a minimal repro and fix with a test

Architecture¶

This topic’s control points and relationships are shown below.

Theory¶

What it is¶

Troubleshooting Python automation is a disciplined ladder: move from symptom to environment to dependencies/APIs to performance, then fix with a test and a note. It combines interpreter skills (sys.executable, venvs), HTTP/cloud error reading, and classic ops habits (what changed, which host, since when).

Why it matters¶

Automation fails in Continuous Integration (CI) at 02:00 with a one-line traceback. Without a method you thrash — reinstalling packages randomly, rotating credentials that were never the issue, or blaming “Python” when the region flag was wrong. A shared ladder shortens Mean Time To Recover (MTTR) and produces repeatable runbooks.

How it works¶

Symptom — exit code, stderr, first occurrence, which job/host/branch.
Environment — sys.executable, Python version, active virtual environment, pinned requirements.
Dependencies / API — import errors, HTTP status, Identity and Access Management (IAM) denials, wrong project/region.
Performance — hang versus slow; CPU, memory, thread-pool pile-up.
Fix — smallest patch, pytest coverage for the bug class, document the gotcha.

Compare which python and sys.prefix before deep debugging; recreate .venv from the lockfile when imports disagree with CI. For signed cloud requests, check clock skew and token expiry. Profile with tracemalloc / cProfile only after the ladder says “performance.”

Key concepts and comparisons¶

Symptom	Likely checks
`ModuleNotFoundError`	Wrong venv; missing install; package typo
Resolver conflict	Pin clash; recreate venv from lock
Works locally, fails CI	Different Python; missing system library
403 from API	IAM/RBAC; wrong account/role
Hang, no logs	Missing timeout; deadlock; waiting on stdin

Production habit	Why
Dry-run / feature flags	Reproduce safely
Temporary log level up	See request IDs without redeploying forever
`run_id` in logs	Correlate CI artefacts
Never ship `breakpoint()`	Jobs freeze waiting for a TTY

Common pitfalls¶

Debugging the system Python while CI uses a venv (or the reverse).
Assuming “network down” when DNS or TLS verify failed.
Fixing symptoms without a regression test.
Adding broad except Exception: pass that hides the next outage.
Rotating every credential before checking region, project, or clock.

Hands-on Lab¶

Focus: practise the core workflow for Troubleshooting Python Automation

mkdir -p ~/rebash-python/module-27
cd ~/rebash-python/module-27

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Step 1 – Fingerprint script¶

cd ~/rebash-python/module-27
source .venv/bin/activate

cat > fingerprint.py << 'EOF'
#!/usr/bin/env python3
from __future__ import annotations

import platform
import sys


def main() -> int:
    print(f"executable={sys.executable}")
    print(f"version={platform.python_version()}")
    print(f"prefix={sys.prefix}")
    print(f"path0={sys.path[0]!r}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
EOF

python fingerprint.py

Step 2 – Simulate missing dep¶

python - <<'PY'
try:
    import totally_missing_rebash_pkg  # noqa: F401
except ModuleNotFoundError as exc:
    print("caught", exc)
    print("fix: install pin or fix venv")
PY

Step 3 – Incident worksheet¶

Create incident.md:

Symptom:
Last good commit:
Python fingerprint:
Command + exit code:
Hypothesis:
Evidence:
Fix + test:

Step 4 – Capstone pointer¶

Review Production DevOps Automation Platform — apply this ladder when the plugin CLI fails in CI.

Validation¶

Lab commands run under ~/rebash-python/module-27/
You can explain each Theory section in your own words
You used modern tooling where it applies to this topic
You can describe one production failure mode for this topic

Code Walkthrough¶

Production practice for Troubleshooting Python Automation always combines:

Inspect before you change (status, plan, logs, dry-run)
Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
Capture evidence (command output, pipeline logs) for handovers
Prefer current tools and APIs over legacy shortcuts
Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations¶

Treat credentials and tokens for python as privileged — never commit them
Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
Validate blast radius before apply/deploy/delete operations
Restrict who can approve production changes
Collect audit logs; limit who can read sensitive traces

Common Mistakes¶

Debugging the system Python while CI uses a venv (or the reverse).

Validate assumptions against the Theory section and official docs before changing production.

Assuming “network down” when DNS or TLS verify failed.

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices¶

Encode Troubleshooting Python Automation changes as code and review them in pull requests
Pin versions (images, modules, actions, provider plugins)
Separate environments with clear promotion gates
Alert on symptoms with runbooks attached
Destroy lab resources; tag everything with owner and expiry where possible