Chaos Testing Redis Failover for Limiters

Every limiter has a degradation mode, and most teams find out what theirs actually is during an incident rather than during a drill. The three experiments here — blackhole, latency injection, and a real failover — take twenty minutes and answer the questions that matter: does the fail-open metric fire, does p99 latency stay bounded, and how much overshoot does a promotion cost? They are the deliberate version of the failure list in rate limit testing and validation.

The problem in concrete numbers

A limiter is configured fail_open: true with a 5-second store timeout. Redis becomes unreachable at 02:14. Every request now waits 5 seconds for a timeout before being admitted, so a service that handled 800 rps with 40 workers exhausts its worker pool in under a second. The rate limiter did not reject a single request — it took the service down by being slow.

The same outage with a 30 ms timeout costs 30 ms per request, the fail-open counter climbs, an alert fires, and traffic flows unlimited but healthy until Redis returns. One configuration line separates the two outcomes, and only a drill reveals which one you have.

Store outage under two timeout budgets With a five-second timeout each request blocks a worker for five seconds and the pool is exhausted almost immediately; with a thirty millisecond timeout the added latency is negligible, the fail-open counter climbs, and the service keeps serving. Same outage, two timeout budgets timeout 5 s every request waits 5 s 40 workers exhausted in < 1 s service unavailable the limiter caused the outage timeout 30 ms +30 ms per request workers keep turning over fail-open metric climbs, alert fires unlimited but alive a limiter must never be able to spend more of the request budget than it saves

The three experiments

Experiment How What it proves Expected result
Blackhole Drop packets to the store port The configured degradation mode is the real one Fail-open admits with a metric, or fail-closed rejects with a 429
Latency injection Add 200–500 ms to store responses The timeout budget bounds request latency Service p99 rises by at most the timeout
Failover Promote a replica How much overshoot a promotion costs Bounded to roughly one burst per key
Memory pressure Fill the store past maxmemory Eviction policy does not delete live limiter keys Keys survive; cache keys evict
Script cache flush SCRIPT FLUSH Client reloads on NOSCRIPT No user-visible errors

Run them in that order: blackhole first because it is the most common failure, then latency, then failover. Memory pressure and script flush are worth adding once the first three pass.

Step-by-step: running the drill

bash
#!/usr/bin/env bash
# drill.sh — three experiments with observable assertions. Staging first.
set -euo pipefail
REDIS_HOST=${REDIS_HOST:?} ; REDIS_PORT=${REDIS_PORT:-6379}
METRICS=${METRICS:-http://localhost:9090/metrics}

metric() { curl -fsS "$METRICS" | awk -v k="$1" '$1 ~ k {print $2; exit}'; }
snapshot() { echo "accepted=$(metric ratelimit_allowed_total) \
failopen=$(metric ratelimit_failopen_total) p99=$(metric http_p99_seconds)"; }

echo "baseline: $(snapshot)"

# --- 1. Blackhole: is the degradation mode what the config claims? -------------
echo "== blackhole 60 s =="
sudo iptables -A OUTPUT -p tcp -d "$REDIS_HOST" --dport "$REDIS_PORT" -j DROP
sleep 60
echo "during:  $(snapshot)"
sudo iptables -D OUTPUT -p tcp -d "$REDIS_HOST" --dport "$REDIS_PORT" -j DROP
sleep 10
echo "after:   $(snapshot)"
# Assertions (fail-open deployment):
#   failopen  MUST have increased      -> the degraded path is instrumented
#   accepted  MUST have kept climbing  -> traffic was admitted, not dropped
#   p99       MUST stay below baseline + timeout

# --- 2. Latency injection: is the timeout budget real? ------------------------
echo "== +300 ms store latency, 60 s =="
sudo tc qdisc add dev eth0 root netem delay 300ms
sleep 60
echo "during:  $(snapshot)"
sudo tc qdisc del dev eth0 root netem
#   p99 MUST rise by at most the configured limiter timeout, not by 300 ms

# --- 3. Failover: how much overshoot does a promotion cost? -------------------
echo "== failover =="
before=$(metric ratelimit_allowed_total)
redis-cli -h "$REDIS_HOST" -p 26379 sentinel failover mymaster
sleep 30
after=$(metric ratelimit_allowed_total)
echo "admitted during failover window: $((after - before))"
#   Compare against the expected rate x 30 s; the excess is the overshoot,
#   and it should be bounded by roughly one burst per active key.
Overshoot around a store failover Before the promotion the accepted rate matches the configured limit, during the promotion a short window admits extra requests as unreplicated state is lost, and after recovery the rate returns to the limit; the excess should be bounded by about one burst per active key. What a promotion costs in admitted requests before: at the limit promotion window state loss = overshoot bounded by one burst after: back at the limit within seconds if the overshoot is unbounded, keys are being recreated empty — check replication and TTLs

Reading the results

Fail-open with no metric change means the degraded path is not instrumented: you were admitting everything and nothing recorded it. Fix the instrumentation before anything else, because this is the failure that stays invisible for months.

p99 rising by the injected latency rather than the timeout means the timeout is not being applied — often because the client library’s timeout only covers connection setup, not command execution. Set a command timeout explicitly.

Overshoot proportional to traffic rather than to key count means keys are being recreated from scratch after promotion, not restored: check that replication is enabled for the limiter database and that TTLs are not shorter than the failover window.

Rejections during a fail-open drill mean the effective policy is not the configured one — commonly because an error path in the middleware treats an exception as a denial. Read the catch block.

Diagnosing a drill from the three signals it produces No change in the fail-open counter means the degraded path is uninstrumented, latency rising by the injected delay means the command timeout is not applied, and overshoot scaling with traffic means limiter keys are being recreated rather than replicated. What each unexpected reading means fail-open flat the degraded path is not instrumented at all fix this before anything else p99 up by 300 ms the timeout covers only connection setup set a command timeout overshoot unbounded keys recreated empty check replication and the eviction policy every one of these is invisible until somebody deliberately breaks the store

Gotchas and edge cases

  • Running the drill without traffic. An idle system degrades beautifully. Generate steady load first, or the drill proves nothing.
  • Blackholing the wrong direction. Dropping only inbound packets lets connections hang differently than a full partition. Drop outbound to the store port for a realistic client-side timeout path.
  • A connection pool that masks the outage. Pre-established connections may fail slowly and unevenly. Run the drill for at least a minute so the pool fully turns over.
  • Sentinel or operator auto-recovery racing the drill. Automated failover may restore service before you finish measuring. Coordinate with the platform team and disable auto-recovery for the window if needed.
  • allkeys-lru on a shared store. Under memory pressure the eviction policy will delete live limiter keys, which looks exactly like a fail-open. Use a dedicated database with volatile-ttl.
  • Forgetting to restore. Leaving an iptables rule or a tc qdisc in place after the drill is the most common way a chaos exercise becomes an incident. Use a trap that always cleans up.

Verification and testing

bash
# Wrap the drill so cleanup always happens, even on failure or interrupt.
cleanup() {
  sudo iptables -D OUTPUT -p tcp -d "$REDIS_HOST" --dport "$REDIS_PORT" -j DROP 2>/dev/null || true
  sudo tc qdisc del dev eth0 root netem 2>/dev/null || true
  echo "cleanup complete"
}
trap cleanup EXIT INT TERM

# After the drill, confirm the limiter is enforcing again — not silently stuck open.
for i in $(seq 1 300); do
  curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: drill_key" \
    https://staging.example.com/v1/ping
done | sort | uniq -c
#  200 x ~limit, 429 for the rest -> enforcement restored
#  300 x 200                      -> STILL failing open: the client never reconnected

That last check is the one people forget. A limiter that fails open correctly but never recovers — because a client library cached a dead connection or a circuit breaker never closed — is an unlimited API that looks perfectly healthy on every dashboard.

Frequently Asked Questions

Should a rate limiter fail open or fail closed?

Per route class. Public read endpoints usually fail open, because unlimited reads for a few minutes is better than an outage. Expensive writes, anything that spends money, and authentication endpoints should fail closed. Whichever you choose, instrument it — an uninstrumented fail-open is indistinguishable from working normally.

What timeout should the store call use?

Tens of milliseconds for a same-region store — enough for a normal round trip plus jitter, far less than your request budget. The rule of thumb: the limiter must never be able to consume more of the request budget than the work it protects.

How much overshoot is acceptable during a failover?

Roughly one burst per active key, because unreplicated state is lost and those keys restart as idle. That is bounded and usually harmless. Overshoot that scales with traffic rather than with key count indicates a deeper problem, such as the limiter database not being replicated at all.

Can these drills run in production?

Yes, on a canary slice, once the staging drill is clean and you have an announced window and a tested rollback. The failover experiment in particular behaves differently at production scale, and the numbers you record are the ones your runbook will need during a real incident.