Site Reliability Engineering

Chaos Engineering for Backend and Infrastructure

Published: July 7, 2024 Reading time: 15 min

Backend chaos engineering assumes you control the runtime — VMs, containers, networks, managed databases — which makes fault injection precise. You choose exactly which pod, zone, or dependency to disrupt. This guide covers steady state for microservices, failure domains from compute to messaging, resilience patterns to validate, Chaos Mesh and Toxiproxy walkthroughs, SLO measurement, CI automation, game-day execution, and tooling. If you are new to the discipline, start with Chaos Engineering: Principles and Practice for the experiment loop, blast-radius controls, and game-day concepts. For how the same mindset applies in the browser, see Chaos Engineering for Frontend Applications. ...

Continue Reading

Chaos Engineering: Principles and Practice

Published: June 6, 2024 Reading time: 9 min

Distributed systems fail in ways unit tests never simulate. A replica set lags behind, a dependency times out under load, a deployment rolls out to half the cluster before someone notices the new health check is wrong. On the client, a payment API returns an empty body after thirty seconds of latency and checkout silently confirms $0.00. Monitoring tells you something broke after the fact. Load tests tell you how the system behaves when everything is working but busy. Chaos engineering asks a sharper question: when this specific component fails, does the rest of the system absorb it? ...

Continue Reading