Reproducible developer environments

This is a general explanation. The experience-backed version is in the case study.

An on-demand environment is a promise: a user asks, and a clean, isolated machine-like workspace appears quickly. Most of the engineering is deciding what that promise costs.

Decide the isolation boundary first

Containers share a kernel, virtual machines do not. Namespaces, virtual clusters and per-environment VMs each place the boundary somewhere different. The boundary determines what a user can safely do inside the environment, including whether they can run containers of their own.

Treat provisioning time as the product metric

Wait time is what users feel. Break it into stages (scheduling, image pull, start-up, readiness) and measure each. Pre-pulled images and warm capacity attack different stages, and you can only choose between them if you know where the time goes.

Make everything declarative

If an environment can only be created by hand, it cannot be recreated after a failure. Terraform for the cluster and Helm for what runs on it mean a rebuild is a command, not a recollection.

Plan the lifecycle, not just the start

Idle timeouts, cleanup and resource limits decide whether the platform stays affordable and fair when many environments share the same nodes.