Skip to content

Containers — Namespaces, cgroups, OverlayFS, and OCI

Overview

Kubernetes nodes are Linux. Container isolation is kernel features, not magic.

This is Tutorial 22 in Module 14: Containers & Cloud of the REBASH Academy Linux for Cloud & DevOps Engineers series — written for administrators, DevOps engineers, SREs, and platform engineers operating production Linux.

Prerequisites

  • SELinux, AppArmor, Fail2Ban, Auditd, and PAM
  • Terminal access with a regular user account (sudo where noted)

Learning Objectives

By the end of this tutorial, you will be able to:

  • Apply the core ideas of “Containers — Namespaces, cgroups, OverlayFS, and OCI” on a real Linux host
  • Use modern tools (ip/ss, systemctl/journalctl) where they apply
  • Complete the lab under ~/rebash-linux/ with clear outputs
  • Relate this topic to Cloud, DevOps, and production operations
  • Explain the failure modes you would check first in an incident

Architecture

Linux ops work sits between humans/automation and the kernel, services, and network. This topic’s control points are shown below.

Architecture diagram for Containers — Namespaces, cgroups, OverlayFS, and OCI

Theory

What it is

Containers are not lightweight VMs; they are ordinary processes given isolated views of the system via namespaces and resource limits via control groups (cgroups), usually with a layered root filesystem such as OverlayFS. The Open Container Initiative (OCI) defines image and runtime specifications so engines (Docker, containerd, CRI-O) can build and run portable images. On Kubernetes, the kubelet talks Container Runtime Interface (CRI) to a runtime that ultimately invokes an OCI runtime such as runc or crun.

Why it matters

Node disk fill, noisy-neighbour CPU, and “it works in Docker but not in the pod” issues are kernel-feature issues underneath the brand name. Understanding namespaces explains network and PID isolation; cgroups explain Out-Of-Memory (OOM) kills and CPU throttling; OverlayFS explains image layer reuse and why deleted image data may still consume space until garbage collection. Ops and SRE roles debug the host as much as the YAML.

How it works

Namespaces isolate PID, network, mount, UTS (hostname), Inter-Process Communication (IPC), user, cgroup, and time views (lsns). cgroup v2 (unified hierarchy under /sys/fs/cgroup) accounts and limits CPU, memory, I/O, and PIDs; systemd and container runtimes both use it. OverlayFS stacks read-only image layers under a writable upper layer for copy-on-write. An OCI image is layers plus config; the runtime creates namespaces/cgroups, sets up the rootfs, and starts the entrypoint. Engines add UX, networking plugins, and image distribution on top of that kernel contract.

Key concepts and comparisons

Mechanism Isolates / limits
Namespaces What the process can see
cgroups What the process can consume
OverlayFS Layered filesystem view
OCI runtime Standard create/start lifecycle
Stack piece Role
Engine (Docker/containerd) Images, API, UX
OCI runtime (runc/crun) Kernel plumbing
Kernel Namespaces, cgroups, OverlayFS

Common pitfalls

  • Treating containers as strong security boundaries without MAC, user namespaces, and least privilege.
  • Ignoring cgroup OOM kills while chasing application “random exits”.
  • Filling the node with image layers and container logs under /var/lib.
  • Debugging only inside the container when lsns/systemd-cgls on the host shows the truth.
  • Assuming PID 1 behaviour inside containers matches a full systemd OS.

Hands-on Lab

Create a workspace for this tutorial.

mkdir -p ~/rebash-linux/lab22 && cd ~/rebash-linux/lab22

Focus: explore lsns/cgroups; sketch OCI stack; inspect a running container if available

Step 1 – Container internals

{
  lsns | head
  cat /proc/self/cgroup
  echo '=== OCI / runtime ==='
  command -v docker && docker info 2>/dev/null | head -n 15
  command -v podman && podman info 2>/dev/null | head -n 10
  command -v crictl && crictl version 2>/dev/null
} 2>&1 | tee container-internals.txt

Final step – Cleanup note

# Keep ~/rebash-linux/ for later tutorials; destroy disposable cloud resources from this lab

Validation

  • Lab commands run under ~/rebash-linux/lab22/
  • You can explain each Theory bullet in your own words
  • You used modern tooling where applicable (ip/ss, systemctl/journalctl)
  • You can describe one production failure mode for this topic

Code Walkthrough

Production Linux practice for Containers — Namespaces, cgroups, OverlayFS, and OCI always combines:

  1. Inspect before you change (status, df, ip, logs)
  2. Prefer reversible, documented changes (config management, drop-ins)
  3. Capture evidence (command output, journal snippets) for handovers
  4. Prefer systemctl/journalctl and ip/ss over legacy tools
  5. Least privilege — escalate with sudo only when required

Keep runbooks short enough to follow at 03:00. Automate the boring checks; keep humans for judgement.

Security Considerations

  • Treat host access and sudo as privileged — audit who can do what
  • Never paste secrets into shell history, tickets, or screenshots
  • Validate device names and paths before destructive disk or rm operations
  • Prefer key-based SSH and deny password auth on internet-facing hosts
  • Collect logs centrally; restrict who can read authentication and audit trails

Common Mistakes

Using legacy networking tools by default

ifconfig/netstat are missing or incomplete on modern images. Fix: use ip and ss.

Editing vendor unit files in place

Package upgrades overwrite /lib/systemd/system. Fix: systemctl edit drop-ins under /etc.

Trusting df without checking inodes and mounts

A full /var or exhausted inodes looks different from root. Fix: df -h, df -i, and findmnt.

Best Practices

  • Golden images + config as code over snowflake hosts
  • Alert on symptoms (failed units, disk, load) with runbooks attached
  • Time-sync (chrony) everywhere — logs and TLS depend on it
  • Separate OS and data volumes on Cloud VMs
  • Practise restore and rescue paths before you need them

Troubleshooting

Symptom Likely cause Fix
Permission denied Mode/owner/ACL/MAC namei -l, id, getfacl, SELinux/AppArmor logs
No route / timeout Routing, DNS, firewall ip route, dig, ss, security groups
Service won’t start Unit/config/deps systemctl status, journalctl -u, config -t
Disk full Logs, containers, deleted-open df/du, lsof +L1, rotate/expand
High load CPU, I/O wait, thrash vmstat, iostat, ps

Summary

Containers — Namespaces, cgroups, OverlayFS, and OCI is essential for Cloud and DevOps engineers operating Linux hosts. Practise the lab until the inspection path is muscle memory, then continue the track.

Interview Questions

  1. How does this topic show up when operating Cloud VMs or Kubernetes nodes?
  2. What would you check first if this area misbehaves in production?
  3. Which modern Linux tools replace older equivalents here?
  4. What security control should accompany this capability?
  5. How would you automate verification of this topic in CI or a cron/timer job?

Sample answer — question 2

Start with blast radius and recent changes, then gather host signals (systemctl --failed, df, ip/ss, journalctl) before making changes. Fix forward with evidence, not guesswork.

References