Troubleshooting AWS¶

Overview¶

Diagnose Amazon Web Services (AWS) production failures with a fixed order: confirm identity and blast radius, then walk the EC2 → IAM → VPC → DNS → storage → Lambda → EKS → cost ladder without random console clicking.

Most “AWS is down” tickets are permission denials, security group / route mistakes, DNS mispoints, or exhausted quotas — not Region-wide outages. Separate control-plane errors (API denied, wrong Region) from data-plane symptoms (timeouts, 5xx, CrashLoop). Capture evidence before you change anything.

Cost hygiene

Debugging can create expensive resources (NAT Gateways, extra load balancers, large CloudWatch Logs Insights queries). Delete temporary helpers and bound log query time ranges.

This is a core tutorial in Module 16 · Troubleshooting of the REBASH Academy AWS for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites¶

Production AWS Landing Zones
Working knowledge of Modules 2–11 (IAM, VPC, compute, storage, containers, serverless, observability)
AWS CLI v2 and read access to the affected account

Learning Objectives¶

By the end of this tutorial, you will be able to:

Start every incident with caller identity, Region, and recent CloudTrail
Triage EC2 instance reachability vs application failure
Decode IAM explicit denies and missing sts:AssumeRole
Trace VPC path: SG → NACL → route → NAT/IGW → endpoint
Isolate DNS (Route 53) vs target health
Debug Lambda and EKS with logs, events, and auth
Use cost spikes as a signal (runaway resources)

Architecture¶

This topic’s control points and relationships are shown below.

Theory¶

What it is¶

Troubleshooting AWS narrows failure domains with a fixed ladder (reorder only when the symptom already proves a layer). Prefer read-only evidence (CloudWatch, CloudTrail, Flow Logs, Health) before mutate.

Step	Focus	First evidence
0	Context	`sts get-caller-identity`, Region, Health, change window
1	EC2	State, status checks, console output, SSM
2	IAM	Explicit Deny, missing Allow, SCP, resource policy
3	VPC	SG, NACL, routes, NAT/endpoints, Flow Logs
4	DNS	Route 53, health checks, resolver, TTL
5	Storage	EBS attach/KMS; S3 403 vs 404
6	Lambda	Timeout, concurrency, VPC, IAM, DLQ
7	EKS	API / nodes / pods; access entries; CNI
8	Cost	Spend spike → runaway capacity, transfer, logs

Why it matters¶

MTTR collapses when you confuse IAM with routing. SRE on-call needs a playbook juniors can follow. SysOps/DevOps exams test “next diagnostic API.” Cost belongs on the ladder — “outage” can be five hundred GPUs or an unbounded Logs Insights bill.

How it works¶

Stabilise: page owners, freeze risky deploys, note start time.
Prove identity, account, Region; define the symptom precisely.
Walk the ladder; never skip IAM on AccessDenied.
Compare working vs broken; check last deploy and CloudTrail.
Smallest reversible fix; validate; write the timeline.

Concept deep dive¶

Method per layer: symptom → evidence → decide → next action.

EC2 failures. describe-instances → describe-instance-status (system vs instance) → console/serial log → SSM reachability → app logs. System check ≈ host/path; instance check ≈ guest OS/disk. Running but unreachable → VPC/IAM (instance profile) before reboot; capture console output first.

IAM issues. Reproduce as the same principal → CloudTrail AccessDenied → identity policy, boundary, SCP, resource policy, session tags → Policy Simulator. Explicit Deny wins; AdministratorAccess still fails under SCP. Cross-account: trust (sts:AssumeRole) and role permissions. Wrong Region often looks like “missing resource.”

VPC connectivity. Source ENI → SG egress → dest SG ingress → NACL (ephemeral ports) → route → NAT/IGW/endpoint → Flow Logs. For ALB: listener → target group → describe-target-health before changing SGs. Private subnet needs NAT or VPC endpoints for public AWS APIs.

DNS problems. dig from the client path → public/private zone → record correctness → Route 53 health/failover → resolver/TTL. Separate “wrong target in DNS” from “DNS right, target unhealthy.”

Storage issues. EBS: state → attachment → KMS usable? → guest mount/disk full. S3: NoSuchKey (404) vs AccessDenied (403) vs wrong Region → bucket policy + IAM + SSE-KMS → Block Public Access → lifecycle. Restores need matching Region/AZ and KMS key.

Lambda failures. Logs → duration vs timeout → throttling/concurrency → execution role → VPC ENI/subnet/SG → event source + DLQ → X-Ray. “Works locally” ≈ missing IAM, VPC egress, or timeout. Check DLQs for async failures.

EKS troubleshooting. Reach API server? → nodes Ready? → kubectl describe/events (Pending, ImagePull, CrashLoop) → capacity/AZ → CNI/SGs → IAM (IRSA/Pod Identity/access entries/aws-auth) → control-plane logs/CloudTrail. Separate control plane, data plane, and workload. Image pulls need ECR auth and network path.

Cost analysis. Cost Explorer by service → account/tag → inventory (ASG, NAT, transfer, log ingestion) → CloudTrail Create* bursts → scale down with change control. Spikes are usually misconfig, not a pricing bug.

Key concepts and comparisons¶

Symptom	Likely layer	First checks
SSM fails, app OK	Instance profile / endpoints	SSM + IAM role
Timeout to ALB	SG / targets / NACL	Target health, Flow Logs
`AccessDenied`	IAM / SCP / resource policy	CloudTrail, simulator
NXDOMAIN / wrong IP	DNS	`dig`, Route 53
Lambda OK locally	IAM / VPC / timeout	Logs, concurrency
Pods Pending	Capacity / CNI / quotas	`kubectl describe`
Bill doubled	Cost / runaway	Cost Explorer

Term	Meaning
Explicit Deny	Wins over Allow
Status checks	EC2 path vs guest OS
Target health	LB view of backends
Quotas	Limits that look like random failure

Common pitfalls¶

Changing SGs before listener/target health.
Ignoring SCPs when AdministratorAccess “fails.”
Wrong Region or account.
Rebooting to “fix” IAM or DNS.
Unbounded Logs Insights creating a cost incident.
Calling every timeout an app bug before Flow Logs/Health.
Mutating before capturing evidence.

Hands-on Lab¶

Create a workspace for this tutorial.

mkdir -p ~/rebash-aws/module-16 && cd ~/rebash-aws/module-16

Focus: practise triage: identity → region → API error → CloudTrail

Step 1 – Triage commands¶

aws sts get-caller-identity
echo "region=$(aws configure get region)"
aws ec2 describe-instances --max-items 1 >/tmp/aws-out.json 2>/tmp/aws-err.txt || true
head -n 20 /tmp/aws-err.txt || true
aws cloudtrail lookup-events --max-results 5 --query 'Events[].{Time:EventTime,Name:EventName}' --output table 2>/dev/null || echo "CloudTrail lookup not permitted"

Step 2 – Incident notes template¶

cat > triage.md << 'EOF'
1. Who am I (STS ARN)? Which region?
2. Exact error code/message
3. Recent CloudTrail events for that API
4. Blast radius / rollback
EOF

Final step – Cleanup note¶

# COST WARNING: prefer describe/list APIs. Destroy anything you create.
# Keep ~/rebash-aws/ for later tutorials

Validation¶

Lab commands run under ~/rebash-aws/module-16/
You can explain each Theory section in your own words
You used modern tooling where it applies to this topic
You can describe one production failure mode for this topic

Code Walkthrough¶

Production practice for Troubleshooting AWS always combines:

Inspect before you change (status, plan, logs, dry-run)
Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
Capture evidence (command output, pipeline logs) for handovers
Prefer current tools and APIs over legacy shortcuts
Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations¶

Treat credentials and tokens for aws as privileged — never commit them
Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
Validate blast radius before apply/deploy/delete operations
Restrict who can approve production changes
Collect audit logs; limit who can read sensitive traces

Common Mistakes¶

Changing SGs before listener/target health.

Validate assumptions against the Theory section and official docs before changing production.

Ignoring SCPs when AdministratorAccess “fails.”

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices¶

Encode Troubleshooting AWS changes as code and review them in pull requests
Pin versions (images, modules, actions, provider plugins)
Separate environments with clear promotion gates
Alert on symptoms with runbooks attached
Destroy lab resources; tag everything with owner and expiry where possible