Troubleshooting Helm¶

Overview¶

Diagnose common Helm failures with a fixed order: lint → template → dry-run → release status → Kubernetes events.

Most “Helm is broken” tickets are template nil pointers, values typos, or cluster admission errors. Separate render failures from apply failures.

This is a core tutorial in Module 12 · Troubleshooting of the REBASH Academy Helm for Kubernetes Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites¶

Production Helm Practices

Learning Objectives¶

By the end of this tutorial, you will be able to:

Fix template nil / type errors
Read helm status / helm history
Recover failed upgrades (--atomic, rollback)
Debug dependency / repo fetch issues

Architecture¶

This topic’s control points and relationships are shown below.

Theory¶

What it is¶

Troubleshooting Helm is a disciplined split between render failures (templates/values/dependencies) and apply failures (Kubernetes API, RBAC, admission, runtime health). Most tickets that say “Helm is broken” are nil pointer template errors, mistyped values keys, repo auth problems, or cluster Events after a successful render. Your job is to locate which layer failed before changing random flags.

Symptom	First checks
Template error	`helm lint`, `helm template --debug`, nil `.Values` path
Install pending/failed	`helm status`, `kubectl get events`, hooks
Upgrade stuck	`--wait` timeout, image pull, PDB, probes
Rollback fails	`helm history`, resource ownership, empty revisions
Dep download fail	repo/OCI URL, credentials, version constraint

Why it matters¶

Mean time to recovery depends on not confusing a bad chart with a bad cluster. Platform on-call needs a fixed order that junior engineers can follow under pressure. The same playbook feeds CI: catch render errors before GitOps ever sees them, and keep --atomic upgrades so failed releases do not leave half-applied state without a path back.

How it works¶

Use this order every time:

Lint — helm lint ./chart.
Render — helm template NAME ./chart -f values-<env>.yaml (add --debug on failure).
Authz — kubectl auth can-i for the deploy identity on required verbs/resources.
Converge carefully — helm upgrade --install --atomic --wait in non-prod first.
Inspect cluster — helm status, kubectl describe, Events on failing objects.
History — helm history; rollback to a known good revision if apply succeeded but behaviour is wrong.
Dependencies — helm dependency build / check repo login if fetch fails.

Separate questions: Did YAML render? Did the API accept it? Did Pods become Ready?

Key concepts and comparisons¶

Failure class	Looks like	Not fixed by
Render	CLI error before resources change	Restarting nodes
Apply / admission	API reject, webhook errors	Editing unrelated values
Runtime	Release deployed, Pods crash	Another `helm template` alone
Fetch	Cannot download chart/deps	Rollback of a different release

Common pitfalls¶

Jumping to helm rollback when the chart never rendered — there may be nothing valid to roll back to.
Ignoring --previous container logs and Events while blaming Helm.
Fixing production with kubectl edit on Helm-owned objects (drift returns on next reconcile).
Assuming --force or deleting release Secrets is a routine fix — both are last resorts with data-loss risk.

Hands-on Lab¶

Create a workspace for this tutorial.

mkdir -p ~/rebash-helm/module-12 && cd ~/rebash-helm/module-12

Focus: Diagnose a failed release using status, hooks, and kubectl

Step 1 – Install a chart then break an upgrade on purpose¶

kubectl create namespace rebash-helm
helm create trouble
helm upgrade --install demo ./trouble -n rebash-helm
# Force a bad image tag to provoke ImagePullBackOff
helm upgrade demo ./trouble -n rebash-helm --set image.repository=nginx --set image.tag=not-a-real-tag-xyz --wait --timeout 45s || true
helm -n rebash-helm status demo || true

Step 2 – Collect evidence and roll back¶

kubectl -n rebash-helm get pods
kubectl -n rebash-helm describe pod -l app.kubernetes.io/instance=demo | sed -n '/Events:/,$p' | head -n 30
helm -n rebash-helm history demo
helm -n rebash-helm rollback demo 1
kubectl -n rebash-helm rollout status deploy -l app.kubernetes.io/instance=demo --timeout=90s || true

Final step – Cleanup note¶

helm uninstall demo -n rebash-helm --ignore-not-found || true
kubectl delete namespace rebash-helm --ignore-not-found
# Workspace kept for notes; remove with: rm -rf "$(pwd)" when finished

Validation¶

Lab commands run under ~/rebash-helm/module-12/
You can explain each Theory section in your own words
You used modern tooling where it applies to this topic
You can describe one production failure mode for this topic

Code Walkthrough¶

Production practice for Troubleshooting Helm always combines:

Inspect before you change (status, plan, logs, dry-run)
Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
Capture evidence (command output, pipeline logs) for handovers
Prefer current tools and APIs over legacy shortcuts
Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations¶

Treat credentials and tokens for helm as privileged — never commit them
Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
Validate blast radius before apply/deploy/delete operations
Restrict who can approve production changes
Collect audit logs; limit who can read sensitive traces

Common Mistakes¶

Jumping to helm rollback when the chart never rendered — there may be nothing valid to r

Validate assumptions against the Theory section and official docs before changing production.

Ignoring --previous container logs and Events while blaming Helm.

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices¶

Encode Troubleshooting Helm changes as code and review them in pull requests
Pin versions (images, modules, actions, provider plugins)
Separate environments with clear promotion gates
Alert on symptoms with runbooks attached
Destroy lab resources; tag everything with owner and expiry where possible