Abstract

Chaos engineering experiments that lack a well-defined steady-state hypothesis generate noise rather than knowledge. The steady-state hypothesis - a measurable assertion about normal system behavior before and after an experiment - is what transforms “we killed a pod and nothing exploded” into a reproducible, falsifiable test of resilience. Codifying hypotheses as executable checks, integrating them with existing observability infrastructure, and automating rollback when hypotheses fail are the practices that elevate chaos engineering from ad-hoc fire drills to systematic reliability testing.

Defining Measurable Steady State

A steady-state hypothesis must be expressed in terms of observable metrics, not subjective impressions. “The service is healthy” is not a hypothesis; “p99 latency for the /checkout endpoint stays below 400ms and error rate remains below 0.5% as measured over a 90-second window” is one. The specificity matters because the post-experiment check uses the same thresholds to determine whether the system recovered, and because quantified hypotheses accumulate into a system characterization over time.

Choosing the right observability signals requires knowing which SLIs actually track user experience. For most web services, this means request latency percentiles and error rates from the gateway or service mesh, not synthetic metrics from within the application. Chaos Toolkit, Steadybit, and Gremlin all support probe definitions that query Prometheus, Datadog, or New Relic as part of the hypothesis check. Integrating directly with the SLO platform - rather than defining separate thresholds - ensures chaos experiment boundaries stay aligned with production reliability targets.

Structuring the Experiment

A well-structured chaos experiment has four sequential phases: the before probe (steady-state check), the method (the chaos action), a wait period, and the after probe (the same steady-state check). If the before probe fails, the experiment aborts without injecting chaos - which itself is a signal that the system was already degraded. If the after probe fails, the experiment reports a deviation and triggers rollback of the chaos action.

Chaos Toolkit’s DSL makes this structure explicit in a JSON or YAML experiment definition. Steadybit and Gremlin provide UI-driven experiment builders that generate the same conceptual structure. The choice of tooling matters less than the discipline of encoding the hypothesis before the experiment runs, not after observing results.

Common Experiment Classes for Distributed Systems

Network partition experiments - blocking traffic between specific services using iptables rules or tc-based packet loss injection - test whether circuit breakers and retries engage as expected. Pod termination experiments validate that Kubernetes rescheduling and service mesh load balancing shed traffic gracefully within the hypothesis window. Resource exhaustion experiments (CPU saturation, memory pressure via stress-ng) reveal whether resource limits, cgroups, and autoscaler responses keep latency within bounds.

Failure injection at the dependency level is often more informative than node-level chaos. Injecting HTTP 503 responses or artificial latency at the boundary of a downstream service with tools like Toxiproxy or Envoy’s fault injection filter tests retry policies and timeout configurations with surgical precision, without requiring privileged kernel access.

Automating Rollback and Experiment Hygiene

Chaos experiments that lack automated rollback are dangerous in continuous integration contexts. Every experiment must have a defined rollback action - re-enabling network rules, restarting terminated processes, removing injected faults - that executes regardless of whether the experiment succeeded or failed. Chaos Toolkit’s rollback mechanism and Steadybit’s attack revert handle this automatically. For bespoke experiments, wrapping chaos actions in try/finally blocks with explicit cleanup ensures the cluster returns to a known state.

Treating chaos experiments as code, reviewed in pull requests alongside application changes, keeps the experiment library coherent with the current system topology and prevents experiments from testing conditions that no longer exist.