Skip to content

Production DNS Operations

Overview

Run DNS changes safely in production: TTL planning, dual answers/failover, change control, and authoritative verification during incidents.

DNS is a control plane. Bad cutovers strand users on old IPs; missing dual-DNS creates SPOFs; ignored TTL makes “instant” failovers lie.

Builds on DNS Fundamentals and DNS Records. Diagram: dns-resolution.

This is a core tutorial in Module 16 · Production Networking of the REBASH Academy Networking for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites

Learning Objectives

By the end of this tutorial, you will be able to:

  • Plan TTL before migrations
  • Verify at authoritative NS, not only public resolvers
  • Design weighted / failover / latency routing concepts
  • Avoid dangling CNAMEs and apex mistakes
  • Document rollback for DNS changes

Architecture

This topic’s control points and relationships are shown below.

Architecture diagram for Production DNS Operations

Theory

What it is

Production Domain Name System (DNS) operations is the discipline of changing and operating DNS safely at scale: Time To Live (TTL) planning, authoritative verification, secondary/dual-provider resilience, health-based failover, and change control. The protocol is the same as fundamentals; the stakes are cutover windows, global caches, and customer traffic.

Why it matters

DNS mistakes are high-blast-radius: a wrong A/AAAA or CNAME can black-hole a region. Caches hold old answers until TTL expires, so “I fixed it” and “users see it” can be hours apart. Registrar or single-provider outages have taken large sites offline. Treating DNS like any other production config — tickets, dual control, rollback — is mandatory for Cloud and Site Reliability Engineering (SRE) teams.

How it works

Before a cutover, lower TTL early enough that clients refresh quickly when you flip records. After publish, query authoritative nameservers with dig @ns … +norecurse so you see zone truth, not a recursive cache. Keep secondaries or dual providers so a single vendor outage is not a company outage. For failover, prefer health-checked records or traffic policies that withdraw bad targets — not manual panic edits at 03:00. Every change needs intent, validation queries, and a rollback (previous record set / Infrastructure as Code commit).

Cloud DNS (Route 53, Azure DNS, Cloud DNS) still needs the same ops habits; the console is not a substitute for TTL maths and authoritative checks.

Key concepts and comparisons

Practice Why
Lower TTL ahead of change Faster client uptake
Query @ns with +norecurse See published truth
Dual providers / secondary Survive registrar/DNS outages
Health-based failover Route away from dead load balancers
Change tickets + dual control Stop fat-finger outages
Check Tooling idea
Auth vs cache dig @authoritative vs default resolver
Propagation patience Respect old TTL still in the wild
Rollback Previous RRset ready before apply

Common pitfalls

  • Flipping production with a multi-hour TTL still cached.
  • Checking only your laptop resolver and declaring global success.
  • Editing live zones without Infrastructure as Code (IaC) or tickets.
  • CNAME at zone apex where the provider forbids it (use ALIAS/ANAME carefully).
  • Leaving debug low TTLs forever (extra query load and flapping risk).

Hands-on Lab

Focus: practise the core workflow for Production DNS Operations

Step 1 – Cutover timeline

T-48h: TTL → 300
T-0: update A/ALIAS; verify authoritative
T+1h: watch metrics; keep old target warm
T+48h: raise TTL if stable

Step 2 – Authoritative check pattern

NS=$(dig example.com NS +short | head -1)
dig @$NS example.com A +norecurse +noall +answer
dig @1.1.1.1 example.com A +noall +answer

Step 3 – Incident card

Write steps for “users hit old IP after cutover” and “NXDOMAIN for prod hostname.”

Validation

  • Lab commands run under ~/rebash-networking//
  • You can explain each Theory section in your own words
  • You used modern tooling where it applies to this topic
  • You can describe one production failure mode for this topic

Code Walkthrough

Production practice for Production DNS Operations always combines:

  1. Inspect before you change (status, plan, logs, dry-run)
  2. Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
  3. Capture evidence (command output, pipeline logs) for handovers
  4. Prefer current tools and APIs over legacy shortcuts
  5. Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations

  • Treat credentials and tokens for networking as privileged — never commit them
  • Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
  • Validate blast radius before apply/deploy/delete operations
  • Restrict who can approve production changes
  • Collect audit logs; limit who can read sensitive traces

Common Mistakes

Flipping production with a multi-hour TTL still cached.

Validate assumptions against the Theory section and official docs before changing production.

Checking only your laptop resolver and declaring global success.

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices

  • Encode Production DNS Operations changes as code and review them in pull requests
  • Pin versions (images, modules, actions, provider plugins)
  • Separate environments with clear promotion gates
  • Alert on symptoms with runbooks attached
  • Destroy lab resources; tag everything with owner and expiry where possible

Troubleshooting

Symptom Likely cause What to do
Stale IP TTL cache Wait; check auth NS; flush local
Split brain Split horizon / lag Compare resolvers; check serial
Cert fail Wrong challenge TXT Query auth for _acme-challenge

Summary

  • DNS changes need TTL theatre and authoritative proof
  • Design for failover and rollback
  • Treat DNS like production config

Interview Questions

  1. How does Production DNS Operations show up when operating Cloud or production platforms?
  2. What would you check first if this area misbehaves in production?
  3. Which modern tools or APIs replace older equivalents here?
  4. What security control should accompany this capability?
  5. How would you automate verification of this topic in CI?

Sample answer — question 2

Start with blast radius and recent changes, gather evidence (logs, status, plan/diff), then fix forward with a known rollback path — not guesswork.

References