You can cut GPU bills by up to 90 per cent, but expect only about 15 seconds to save work when a node is reclaimed. Google Cloud published a how-to for running interruptible, production AI workloads on Google Kubernetes Engine, showing engineers how to catch the Kubernetes SIGTERM, persist checkpoints to Cloud Storage and make startup logic idempotent so interrupted jobs resume cleanly. The guidance treats these practices as the baseline for cost‑optimised, interruptible training, not an experimental hack.

Why gamble on Spot and preemptible capacity when the whole point of training is long uninterrupted runs? Google Cloud's answer is blunt: accept the gamble, then design your software so the gamble can't break anything.

Handle the signal, save the state

The core of Google Cloud's guidance is operational and low-level. When a Spot VM node is reclaimed the host issues an ACPI signal that Kubernetes converts into a SIGTERM for containers. Non-system pods get a short grace period, which Google Cloud documents as up to 15 seconds on GKE, before the system escalates to SIGKILL. That tiny window determines what a shutdown handler can realistically finish.

Practically, applications must do four things inside that handler: stop accepting new batches, flush in-memory metrics and any transient state, write a final checkpoint, and exit with a success status so orchestration and retry logic don't treat the pod as failed. The guide pairs that shutdown practice with periodic external checkpointing: save model weights, optimizer state and training counters to Cloud Storage so a restarted container can pick up where another left off.

There are two specific operational calls in the recommendation. First, put buckets in the same region as the cluster to reduce latency and to avoid outbound egress fees; the guidance even cites regional co-location examples such as us-central1 for nearby clusters. Second, make container startup idempotent: start-up scripts should attempt to load the latest checkpoint and resume, not create a fresh run that duplicates work or corrupts state.

Mastheads

Making the application resilient is necessary but not enough. Google Cloud frames these patterns as part of a larger platform engineering checklist: design idempotent operations so retries are safe, use persistent external storage for critical state, and pick a checkpoint cadence that balances IO overhead against the amount of lost work you can tolerate.

Those recommendations line up with recent Kubernetes resilience guidance that emphasises Pod Disruption Budgets, readiness and liveness probes, autoscaling, node-pool affinity rules and multi-zone clusters.

Cost math is the lever that pushes teams down this path. Spot-style instances can be as much as 90 per cent cheaper than pay-as-you-go pricing, a saving that explains why teams accept intermittent compute. The trade-off is availability. Eviction-notice windows differ across clouds and they shape what you can finish in a shutdown handler. Google Cloud documents up to 15 seconds for non-system pods on GKE. Those timing differences influence checkpoint cadence and how you partition work across batch sizes and task length.

Platform engineers at scale are also experimenting higher up the stack. Large AI users are adding scheduling and slice-management layers to isolate faulty hardware quickly and to re-balance capacity without full node recreate cycles. More sophisticated schedulers and modular compute designs, including smarter bin-packing for accelerator fleets, reduce the window of lost work while keeping utilisation high.

There is a clear counter-argument: buy reserved or dedicated GPUs and avoid interruptions altogether. That path reduces engineering complexity but comes at a predictability premium. The choice is a trade-off: teams that want the cheapest compute pay the integration tax for resilience. Google Cloud's materials push that tax into reproducible engineering patterns so the cost decision is operational, not experimental.

For platform teams that choose interruptible pools, Google Cloud pairs application rules with cluster-level protections. Use Pod Disruption Budgets to limit voluntary maintenance impact, spread node pools across zones to avoid correlated failure, and use affinity rules to place critical services on non-evictable nodes. Autoscaling and dynamic workload schedulers can queue scarce GPUs and place training jobs on spare Spot capacity, while keeping higher-priority workloads on reserved instances.

Finally, operational detail matters: choose checkpoint frequency deliberately, monitor the time-to-write for your checkpoints in the cluster region, and test SIGTERM handlers under real eviction scenarios. The how-to treats these steps as engineering hygiene for cost‑optimised ML, not optional hardening.

Related Articles

Buying reserved or dedicated GPUs reduces engineering complexity but increases costs. The single concrete constraint that will shape every design here is the eviction-notice window: Google Cloud documents up to 15 seconds of grace for non-system pods on GKE, a hard ceiling that decides how much shutdown work you can reliably complete. Teams must now weigh the potential savings against that operational cost and choose checkpoint cadences and architectures that match their loss tolerance and budget.

This article was created with AI assistance.