Skip to content

Lab — Python Secrets Scanner

Lab Overview

Purpose: Detect accidental secrets in source trees before push.

Scenario: A webhook URL and AWS-like key were committed last quarter. You need a lightweight scanner for pre-commit and CI.

Expected outcome: CLI walks files with pathlib, matches regex rules, exits 1 on findings; allowlist support.

This is a lab, not a tutorial

Apply Security for DevOps Python and File Handling.

Business Scenario

Secret scanning SaaS is coming next quarter. Until then, a focused Python scanner covers the top patterns.

Learning Objectives

  • Walk trees with pathlib; skip .git, .venv, binaries
  • Compile secret regexes (AKIA…, PEM headers, Slack webhooks)
  • Support allowlist file for false positives
  • Never print full secret values (truncate)

Prerequisites

Knowledge

Software

Python stdlib. Estimated cost: £0.

Environment

mkdir -p ~/rebash-lab-python-secrets/{sample/{ok,bad},out}
cd ~/rebash-lab-python-secrets
python3 -m venv .venv && source .venv/bin/activate

Initial State

echo 'api_url: https://example.com' > sample/ok/config.yaml
printf 'AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n' > sample/bad/leaked.env
printf '-----BEGIN RSA PRIVATE KEY-----\nMIIE...\n-----END RSA PRIVATE KEY-----\n' > sample/bad/id_rsa

Task

Create secrets_scan.py --path sample writing out/findings.json. Exit 0 on clean, 1 on findings. Redact matches to first/last 4 characters.

Validation

python secrets_scan.py --path sample/ok; echo $?    # 0
python secrets_scan.py --path sample/bad; echo $?   # 1
  • Detects AKIA pattern and PEM header
  • Findings redact secret bodies
  • Skips itself if you point at .venv

Troubleshooting

Symptom Fix
Binary noise Skip non-text via charset / extension allowlist
False positive --allowlist allowlist.txt with path substrings

Cleanup

deactivate 2>/dev/null || true
rm -rf ~/rebash-lab-python-secrets

Production Discussion

Use gitleaks/trufflehog for depth; keep custom rules for org-specific tokens. Rotate any secret the scanner finds — assume compromise.