Skip to content

Remote State and Backends

Overview

Explain why remote state and locking matter, compare S3 / Azure / GCS and HCP Terraform patterns, and practise an explicit local backend path as a safe stepping stone.

Local state cannot support teams. Concurrent applies corrupt files; laptops get wiped; pull requests lack a single source of truth. Remote backends provide shared durable storage, locking, encryption, and access control. Providers still talk to cloud APIs; the backend stores state.

This is a core tutorial in Module 8 · State Management of the REBASH Academy Terraform for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites

Learning Objectives

By the end of this tutorial, you will be able to:

  • Explain shared storage, locking, and encryption for team state
  • Compare local, S3, Azure, GCS, and HCP Terraform / cloud backends
  • Describe terraform init -migrate-state and partial -backend-config
  • Use terraform_remote_state cautiously

Architecture

This topic’s control points and relationships are shown below.

Remote backends

Theory

What it is

A backend is the storage and locking engine for state. The default is local (a file on disk). Remote backends move that file into object storage or a managed service so every engineer and CI runner sees the same bindings. Team requirements: shared durable storage, mutual exclusion (locking), encryption at rest and in transit, access control and audit, and versioning / backups for recovery.

Why it matters

Two engineers applying without locks can corrupt state or apply conflicting plans. An open state bucket is a credential disclosure waiting to happen — state often holds passwords, keys, and connection strings. Platform teams standardise one backend pattern (bucket + lock table, or HCP Terraform) so product roots inherit security defaults instead of inventing them per project.

How it works

  1. Declare a backend block inside terraform { } (or a cloud block for HCP Terraform — mutually exclusive with backend).
  2. terraform init configures the backend; changing type or key often needs -migrate-state or -reconfigure.
  3. Plans and applies read/write remote state under a lock for the duration of the operation.
  4. CI commonly uses partial backend config: commit a skeleton; pass region, role, or table via -backend-config.
  5. data "terraform_remote_state" reads outputs from another state’s key — convenient but coupling; prefer few stable outputs or a real data plane (Parameter Store, service discovery).

Cloud patterns (study shape — wire credentials only with a real account): S3 (versioned bucket + lock table / native locks, encrypt = true), AzureRM (storage container + blob lease), GCS (bucket + prefix), HCP Terraform (managed state and locks).

Key concepts and comparisons

Concern Local Remote
Storage Disk file Object store / HCP
Locking None (file races) Native / DynamoDB / platform
Sharing Copy files (bad) IAM / team permissions
Fit Solo labs Teams and CI

Migrate deliberately: add backend → init -migrate-state → verify state list → delete local copies only after remote is authoritative.

Common pitfalls

  • Remote state without locking — concurrent apply corruption.
  • World-readable state buckets or loose ACLs.
  • One giant state for all environments — blast radius and lock contention.
  • Deep terraform_remote_state webs instead of stable contracts.
  • Embedding long-lived access keys in backend config — use roles / OIDC.

Hands-on Lab

Create a workspace for this tutorial.

mkdir -p ~/rebash-terraform/module-08/remote-backends/{state,state-b,out} && cd ~/rebash-terraform/module-08/remote-backends/{state,state-b,out}

Focus: Configure a local backend path to practise backend blocks (no cloud account)

Step 1 – Move state into an explicit local backend directory

mkdir -p state-backend
cat > main.tf <<'EOF'
terraform {
  required_providers {
    local = { source = "hashicorp/local", version = "~> 2.5" }
  }
  backend "local" {
    path = "state-backend/terraform.tfstate"
  }
}
resource "local_file" "marker" {
  content  = "remote-style-local-backend
"
  filename = "${path.module}/marker.txt"
}
EOF
terraform init
terraform apply -auto-approve

Step 2 – Verify state location and discuss locking

ls -la state-backend/terraform.tfstate
terraform state list
echo "S3/Azure/GCS backends add shared storage + locking; this lab only relocates local state"

Final step – Cleanup note

terraform destroy -auto-approve
# Workspace kept for notes; remove with: rm -rf "$(pwd)" when finished

Validation

  • Lab commands run under ~/rebash-terraform/module-08/remote-backends/{state,state-b,out}/
  • You can explain each Theory section in your own words
  • You used modern tooling where it applies to this topic
  • You can describe one production failure mode for this topic

Code Walkthrough

Production practice for Remote State and Backends always combines:

  1. Inspect before you change (status, plan, logs, dry-run)
  2. Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
  3. Capture evidence (command output, pipeline logs) for handovers
  4. Prefer current tools and APIs over legacy shortcuts
  5. Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations

  • Treat credentials and tokens for terraform as privileged — never commit them
  • Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
  • Validate blast radius before apply/deploy/delete operations
  • Restrict who can approve production changes
  • Collect audit logs; limit who can read sensitive traces

Common Mistakes

Remote state without locking — concurrent apply corruption.

Validate assumptions against the Theory section and official docs before changing production.

World-readable state buckets or loose ACLs.

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices

  • Encode Remote State and Backends changes as code and review them in pull requests
  • Pin versions (images, modules, actions, provider plugins)
  • Separate environments with clear promotion gates
  • Alert on symptoms with runbooks attached
  • Destroy lab resources; tag everything with owner and expiry where possible

Troubleshooting

Symptom Likely cause Fix
Auth / permission denied Wrong identity, policy, or scope Check caller identity, roles, and least-privilege policies
Timeout / no route Network, DNS, security group, or endpoint Trace path, DNS, and allow-lists before retrying
Drift / unexpected plan Manual change or wrong state/workspace Reconcile desired vs actual; avoid click-ops on managed resources
Pipeline/job red Flaky step, cache, or missing secret Read failing step logs; bisect recent workflow/config changes
Cost spike Idle load balancer, NAT, oversized compute Inventory billable resources; stop/delete labs promptly

Summary

Remote State and Backends is essential for Cloud and DevOps engineers working with terraform. Practise the lab until the inspection and change path is muscle memory, then continue the track.

Interview Questions

  1. What problems do remote backends solve?
  2. What is state locking and why does it matter?
  3. How does partial apply failure interact with remote state?
  4. What security controls belong on a remote state bucket?
  5. When might you split state across multiple backends/workspaces?

Sample answer — question 2

Locking prevents two applies from corrupting state or racing changes. Without locks, concurrent runs can overwrite each other’s state snapshots.

Sample answer — question 4

Encrypt the bucket, block public access, limit IAM to CI roles, enable versioning, and audit access. State is as sensitive as production config.

References