Production incident drills

A malformed data payload is serialized into a shared global cache. Every microservice pod that fetches it encounters an infinite memory-allocation bug during deserialization, triggering an instant kernel OOM-kill. As Kubernetes replaces the dead pods, they immediately pull the same key and die, paralyzing the cluster.

A high-traffic cache key expires, causing a massive wave of concurrent requests to slam the primary database. Manage database connection exhaustion and implement protective application patterns under load.

Services suddenly cannot resolve internal hostnames after a platform change. Trace the blast radius before rebooting everything.

A deployment rolls out and new pods never become ready. Diagnose whether it's config, resources, or upstream dependency.

Read traffic is hitting a stale replica while writes pile up on the primary. Decide how you triage without making lag worse.

Mobile clients and webhooks fail TLS handshake after a cert lapsed on the edge. Decide renewal vs rollback under pressure.