Skip to content

Production Pipelines and Environments

Overview

Design environment promotion (dev → staging → production) with protected environments, manual approvals, rollback paths, progressive delivery, and feature-flag controls.

Production CI/CD is controlled promotion of an immutable artefact through named environments, not “run deploy on every push to main”. GitLab environments, protection rules, and when: manual jobs encode who may promote and how you recover.

This is a core tutorial in Module 15 · Production Pipelines of the REBASH Academy GitLab CI/CD for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.

Prerequisites

Learning Objectives

By the end of this tutorial, you will be able to:

  • Model env promotion with environment: and sequential gates
  • Protect production with approvals and deployment permissions
  • Document rollback (previous version / previous release)
  • Outline progressive delivery and feature-flag separation

Architecture

This topic’s control points and relationships are shown below.

GitLab production

Theory

What it is

A production pipeline promotes the same build (image digest, package version) across environments — logical targets such as development, staging, and production. GitLab tracks deployments per environment, supports manual jobs, and (on eligible tiers) protected environments that restrict who can deploy. Progressive delivery reduces blast radius (canary, blue/green, traffic shifting). Feature flags decouple deploy (ship binary) from release (enable behaviour).

Control Intent
Environment name Track URL, status, history
Manual job Human promote / confirm
Protected environment Role / approval gate
Immutable artefact Same SHA/digest every stage
Feature flag Runtime expose without redeploy

Why it matters

Auto-deploying every merge to production maximises speed and incident rate. Enterprises need auditable promotion: who approved, which digest is live, and how to roll back in minutes. Progressive delivery and flags let platform teams ship continuously while product controls exposure — essential for multi-tenant cloud services and regulated workloads.

How it works

  1. Build once — tag image/package with commit SHA or SemVer; never rebuild for prod.
  2. Deploy to non-prod automatically after tests; run smoke checks.
  3. Staging — optional manual or auto; integration and acceptance tests against staging URLs.
  4. Production — when: manual and/or protected environment approvals; deploy the same digest.
  5. Verify — health checks, dashboards, error budgets; keep the previous revision ready.
  6. Rollback — redeploy previous digest / helm rollback / traffic shift back; document the job.
  7. Flags — risky features default off; enable gradually after deploy is healthy.

Keep production variables and runners scoped; production jobs should not share broad credentials with MR pipelines.

Key concepts and comparisons

Pattern Blast radius Complexity
All-at-once High Low
Blue/green Medium (cutover) Medium
Canary / progressive Low (ramp traffic) Higher
Feature flags Behaviour-level App + ops process
Continuous Delivery Continuous Deployment
Ready to prod; human may gate Auto to prod when green

Common pitfalls

  • Rebuilding the image in the production job — digests diverge from what staging tested.
  • Unprotected production environment — any Developer can click deploy.
  • Rollback untested until an outage — practice in staging.
  • Using feature flags as a substitute for broken deploy pipelines — flags hide, they do not fix bad artefacts.

Hands-on Lab

Create a workspace for this tutorial.

mkdir -p ~/rebash-gitlab/module-15 && cd ~/rebash-gitlab/module-15

Focus: model staging/production environments with manual production gate

Step 1 – Environment-aware deploy jobs

cat > .gitlab-ci.yml << 'EOF'
stages: [build, deploy]
build:
  stage: build
  image: alpine:3.20
  script: ["mkdir -p dist && echo $CI_COMMIT_SHA > dist/REVISION"]
  artifacts: {paths: [dist/]}
deploy_staging:
  stage: deploy
  image: alpine:3.20
  environment: {name: staging, url: https://staging.example.invalid}
  script: ["test -f dist/REVISION", "echo deploy staging"]
deploy_production:
  stage: deploy
  image: alpine:3.20
  environment: {name: production, url: https://www.example.invalid}
  needs: [deploy_staging]
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script: ["echo production requires manual approval"]
EOF

Step 2 – Confirm environment blocks

grep -A3 'environment:' .gitlab-ci.yml
grep -A2 'when: manual' .gitlab-ci.yml

Final step – Cleanup note

# Keep ~/rebash-gitlab/ for later tutorials

Validation

  • Lab commands run under ~/rebash-gitlab/module-15/
  • You can explain each Theory section in your own words
  • You used modern tooling where it applies to this topic
  • You can describe one production failure mode for this topic

Code Walkthrough

Production practice for Production Pipelines and Environments always combines:

  1. Inspect before you change (status, plan, logs, dry-run)
  2. Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
  3. Capture evidence (command output, pipeline logs) for handovers
  4. Prefer current tools and APIs over legacy shortcuts
  5. Least privilege — escalate credentials only when required

Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.

Security Considerations

  • Treat credentials and tokens for gitlab as privileged — never commit them
  • Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
  • Validate blast radius before apply/deploy/delete operations
  • Restrict who can approve production changes
  • Collect audit logs; limit who can read sensitive traces

Common Mistakes

Rebuilding the image in the production job — digests diverge from what staging tested.

Validate assumptions against the Theory section and official docs before changing production.

Unprotected production environment — any Developer can click deploy.

Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.

Changing production without a rollback path

Always know how to revert (previous artefact, prior release, state rollback, DNS failback).

Best Practices

  • Encode Production Pipelines and Environments changes as code and review them in pull requests
  • Pin versions (images, modules, actions, provider plugins)
  • Separate environments with clear promotion gates
  • Alert on symptoms with runbooks attached
  • Destroy lab resources; tag everything with owner and expiry where possible

Troubleshooting

Symptom Likely cause Fix
Auth / permission denied Wrong identity, policy, or scope Check caller identity, roles, and least-privilege policies
Timeout / no route Network, DNS, security group, or endpoint Trace path, DNS, and allow-lists before retrying
Drift / unexpected plan Manual change or wrong state/workspace Reconcile desired vs actual; avoid click-ops on managed resources
Pipeline/job red Flaky step, cache, or missing secret Read failing step logs; bisect recent workflow/config changes
Cost spike Idle load balancer, NAT, oversized compute Inventory billable resources; stop/delete labs promptly

Summary

Production Pipelines and Environments is essential for Cloud and DevOps engineers working with gitlab. Practise the lab until the inspection and change path is muscle memory, then continue the track.

Interview Questions

  1. How do GitLab environments help track deployments?
  2. Why make production deploy when manual even if staging is automatic?
  3. What should stop an automatic promote from staging to production?
  4. How do protected environments reduce risk?
  5. What evidence do you keep for a production change?

Sample answer — question 2

Check environment name/url, deployment job rules, and whether the commit is on the allowed branch. Read the job log and app health next.

Sample answer — question 4

Limit who can run production jobs, require approvals on protected environments, and inject production secrets only into those jobs.

References