Platform Engineering on Kubernetes¶
Overview¶
Explain CRDs and Operators as the extension model, outline admission webhooks, and design namespace-based multi-tenancy with quotas and RBAC.
Platform teams expose paved roads: templates, Operators (extend the API), policy (OPA/Kyverno via admission), and self-service namespaces.
This is a core tutorial in Module 16 · Platform Engineering of the REBASH Academy Kubernetes for Cloud & DevOps Engineers series — written for Cloud, DevOps, Platform, and SRE engineers.
Prerequisites¶
Learning Objectives¶
By the end of this tutorial, you will be able to:
- Define CRD vs custom controller / Operator
- Place validating/mutating admission
- Sketch soft multi-tenancy (ns + quota + RBAC)
- List what not to put in a shared cluster
Architecture¶
This topic’s control points and relationships are shown below.
Theory¶
What it is¶
Platform engineering on Kubernetes means building paved roads so product teams ship without becoming full-time cluster experts. The extension model is the Operator pattern: Custom Resource Definitions (CRDs) add new API types; controllers reconcile those types into Pods, buckets, databases, or policies. Admission webhooks (mutating/validating) and policy engines (Kyverno, OPA Gatekeeper) enforce standards at create time. Multi-tenancy usually starts with namespaces, RBAC, and quotas.
Why it matters¶
Handing every team a raw cluster creates snowflake YAML and security drift. A platform productises golden paths: templates, service catalogues, shared ingress/observability, and self-service namespaces. Operators encode operational knowledge (backup, failover) into software that reconciles continuously — the same control-loop idea as Deployments, applied to higher-level services.
How it works (mental model)¶
- Define a CRD (
Widget) describing desired intent. - An Operator watches Widget objects and creates Deployments, Services, PVCs, etc.
- Admission policies validate labels, block
:latest, or inject sidecars. - Tenants get a namespace with RoleBindings and ResourceQuotas; platform owns cluster add-ons.
- GitOps delivers both platform components and tenant apps.
If reconciliation fails, the custom resource shows conditions — debug like any controller.
Key concepts / comparisons¶
| Concept | Meaning |
|---|---|
| CRD | Schema for a custom API object |
| Controller / Operator | Reconcile custom resources |
| Admission | Mutate/validate before persist |
| Soft multi-tenancy | Shared cluster, ns isolation |
| Hard multi-tenancy | Separate clusters / stronger isolation |
| Shared cluster OK | Prefer isolation |
|---|---|
| Stateless apps with NetworkPolicy | Untrusted code execution |
| Internal tools | Hostile multi-tenant SaaS without extra controls |
Common pitfalls¶
- Building Operators before documenting the paved path — golden paths beat custom CRDs for many apps.
- Cluster-admin bindings for every tenant “to unblock”.
- Admission webhooks that deadlock the API (fail closed without care during outages).
- CRDs without status/conditions — users cannot see why reconcile stalled.
- Assuming namespaces equal security isolation without NetworkPolicy and PSA.
Hands-on Lab¶
Create a workspace for this tutorial.
Focus: Package a golden-path Deployment template teams can reuse
Step 1 – Create a reusable base manifest set¶
kubectl create namespace rebash-lab
mkdir -p platform/base
cat > platform/base/deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: TEAM_APP
namespace: rebash-lab
labels:
platform.rebash.ai/tier: app
spec:
replicas: 2
selector:
matchLabels:
app: TEAM_APP
template:
metadata:
labels:
app: TEAM_APP
spec:
containers:
- name: app
image: nginx:1.27-alpine
resources:
requests:
cpu: 50m
memory: 64Mi
EOF
sed 's/TEAM_APP/payments/g' platform/base/deployment.yaml > payments.yaml
kubectl apply -f payments.yaml
kubectl -n rebash-lab rollout status deploy/payments
Step 2 – Validate platform labels and self-service checklist¶
kubectl -n rebash-lab get deploy payments --show-labels
kubectl -n rebash-lab get deploy payments -o jsonpath='{.spec.template.spec.containers[0].resources}{"
"}'
cat > platform/CHECKLIST.md <<'EOF'
Golden path: namespace, requests/limits, labels, probes, non-root (next iteration).
EOF
Final step – Cleanup note¶
kubectl delete namespace rebash-lab --ignore-not-found
# Workspace kept for notes; remove with: rm -rf "$(pwd)" when finished
Validation¶
- Lab commands run under
~/rebash-k8s/module-16/ - You can explain each Theory section in your own words
- You used modern tooling where it applies to this topic
- You can describe one production failure mode for this topic
Code Walkthrough¶
Production practice for Platform Engineering on Kubernetes always combines:
- Inspect before you change (status, plan, logs, dry-run)
- Prefer reversible, documented changes (Git, IaC, drop-ins, version pins)
- Capture evidence (command output, pipeline logs) for handovers
- Prefer current tools and APIs over legacy shortcuts
- Least privilege — escalate credentials only when required
Keep runbooks short enough to follow under pressure. Automate checks; keep humans for judgement.
Security Considerations¶
- Treat credentials and tokens for kubernetes as privileged — never commit them
- Prefer short-lived auth (OIDC, roles, SSO) over long-lived keys
- Validate blast radius before apply/deploy/delete operations
- Restrict who can approve production changes
- Collect audit logs; limit who can read sensitive traces
Common Mistakes¶
Building Operators before documenting the paved path — golden paths beat custom CRDs for m
Validate assumptions against the Theory section and official docs before changing production.
Cluster-admin bindings for every tenant “to unblock”.
Lab shortcuts (open security groups, admin roles, skip approvals) must not ship unchanged.
Changing production without a rollback path
Always know how to revert (previous artefact, prior release, state rollback, DNS failback).
Best Practices¶
- Encode Platform Engineering on Kubernetes changes as code and review them in pull requests
- Pin versions (images, modules, actions, provider plugins)
- Separate environments with clear promotion gates
- Alert on symptoms with runbooks attached
- Destroy lab resources; tag everything with owner and expiry where possible
Troubleshooting¶
| Symptom | Likely cause | Fix |
|---|---|---|
| Auth / permission denied | Wrong identity, policy, or scope | Check caller identity, roles, and least-privilege policies |
| Timeout / no route | Network, DNS, security group, or endpoint | Trace path, DNS, and allow-lists before retrying |
| Drift / unexpected plan | Manual change or wrong state/workspace | Reconcile desired vs actual; avoid click-ops on managed resources |
| Pipeline/job red | Flaky step, cache, or missing secret | Read failing step logs; bisect recent workflow/config changes |
| Cost spike | Idle load balancer, NAT, oversized compute | Inventory billable resources; stop/delete labs promptly |
Summary¶
Platform Engineering on Kubernetes is essential for Cloud and DevOps engineers working with kubernetes. Practise the lab until the inspection and change path is muscle memory, then continue the track.
Interview Questions¶
- What is a golden path in platform engineering?
- How do templates or Helm charts reduce cognitive load for product teams?
- What should a platform expose as self-service versus keep as a ticket?
- How do you prevent golden paths from becoming unchangeable constraints?
- Which Kubernetes APIs commonly underpin an internal developer platform?
Sample answer — question 2
Golden paths encode defaults for Deployments, networking, observability, and security so teams ship without reinventing cluster details.
Sample answer — question 4
Offer escape hatches, versioned templates, and feedback loops. Rigid platforms that block legitimate needs drive shadow IT; measure adoption and iterate with users.