Fail-Open vs Fail-Closed on Redis Outage

Every limiter has a third outcome besides allow and deny: cannot decide. What happens then is a policy choice, and leaving it to a default — or to whatever an exception handler happens to do — is how an outage in a counter store becomes either an unlimited API or a total one. This is the degradation half of the Redis counter architecture guide.

The problem in concrete numbers

Redis becomes unreachable for 90 seconds during a failover that took longer than expected.

Fail-open, correctly implemented: every request is admitted, the ratelimit_failopen_total counter climbs at 800 per second, an alert fires within 30 seconds, and the backend absorbs unlimited traffic for a minute and a half. If your service can survive that, this is the right answer for public read endpoints.

Fail-closed: every request returns 429, clients back off, the backend is idle, and 72,000 requests are rejected. For a payments endpoint that is correct — better to reject a charge than to process it without a quota check.

Neither, which is what most systems ship: a 5-second client timeout with no policy, so every request blocks a worker for 5 seconds, the pool exhausts in under a second, and the service returns 502 for everything including endpoints that have no limiter at all.

Choosing per route class

The right answer is not one policy for the whole API. It is a default — usually fail-open — with an explicit list of route classes that override it, reviewed whenever a new endpoint moves money.

The middle option: local fallback

Neither extreme is necessary if the process can enforce something while the store is away. A small in-memory bucket per node, sized at the fleet limit divided by the node count, keeps a bound on the damage without rejecting everything.

// limiter.ts — shared store first, local bucket as a bounded fallback.
import Redis from "ioredis";

const redis = new Redis(process.env.REDIS_URL!, {
  commandTimeout: 30,              // hard bound: the limiter cannot own the request budget
  maxRetriesPerRequest: 1,
  enableOfflineQueue: false,       // fail fast instead of queueing during an outage
});

type Policy = "open" | "closed" | "local";

interface Verdict { allowed: boolean; degraded: boolean; retryAfterMs: number; }

// One in-memory bucket per key per node, used only while the store is unavailable.
const local = new Map<string, { tokens: number; ts: number }>();

function localCheck(key: string, capacity: number, refillPerSec: number, nodes: number): boolean {
  // Each node enforces its share, so the fleet stays near the intended total.
  const cap = Math.max(1, Math.floor(capacity / nodes));
  const rate = refillPerSec / nodes;
  const now = Date.now();
  const s = local.get(key) ?? { tokens: cap, ts: now };
  s.tokens = Math.min(cap, s.tokens + ((now - s.ts) / 1000) * rate);
  s.ts = now;
  if (s.tokens < 1) { local.set(key, s); return false; }
  s.tokens -= 1; local.set(key, s);
  return true;
}

export async function check(
  key: string, capacity: number, refillPerSec: number,
  { policy, nodes = 1 }: { policy: Policy; nodes?: number },
): Promise<Verdict> {
  try {
    const [allowed, , retryMs] = await redis.evalsha(SHA, 1, key, capacity, refillPerSec) as number[];
    return { allowed: allowed === 1, degraded: false, retryAfterMs: retryMs };
  } catch (err) {
    // The store could not decide. Everything below is the POLICY, not an accident.
    metrics.increment("ratelimit_degraded_total", { policy, key_class: classify(key) });
    switch (policy) {
      case "closed":
        return { allowed: false, degraded: true, retryAfterMs: 1000 };
      case "local":
        return { allowed: localCheck(key, capacity, refillPerSec, nodes), degraded: true, retryAfterMs: 1000 };
      case "open":
      default:
        return { allowed: true, degraded: true, retryAfterMs: 0 };
    }
  }
}

Two details in that code carry most of the value. commandTimeout: 30 bounds how much of the request budget the limiter can consume — without it, the policy never runs because every request is still waiting. And enableOfflineQueue: false prevents the client from silently queueing thousands of commands that will all resolve at once when the store returns, producing a second spike right after recovery.

Step-by-step: implementing the policy

• Write the default policy down, per route class, and put it in configuration rather than in a catch block.
• Set a command timeout in the tens of milliseconds so the policy actually executes.
• Disable offline queueing so commands do not pile up and flood the store on recovery.
• Add a circuit breaker: after N consecutive failures, stop calling the store for a cooldown and apply the policy immediately.
• Emit ratelimit_degraded_total labelled with the policy and route class on every degraded decision.
• Alert on that counter being non-zero for more than a few seconds — not on its rate.
• For fail-closed routes, return 429 with a Retry-After, never a 500; the client behaviour you want is a back-off.
• Drill it: blackhole the store in staging and confirm the observed behaviour matches the configured one.

Gotchas and edge cases

• The exception handler is the policy. A try/catch that logs and continues is a fail-open decision nobody made. Make it explicit and configurable.
• Timeouts that only cover connection setup. Many clients distinguish connect timeout from command timeout; a stalled but connected store then blocks indefinitely. Set both.
• Recovery stampede. When the store returns, every queued command fires and every client retries at once. Disable offline queues, and jitter Retry-After on the fail-closed path.
• Fail-closed returning 500. A 500 tells clients to retry immediately and tells your error budget it was a server fault. Use 429 with a short Retry-After.
• No distinction between “store down” and “over limit”. They need different metrics and different alerts; a spike in one is an infrastructure page, the other is business as usual.
• Local fallback with no node awareness. A per-node bucket sized at the full fleet limit multiplies the effective limit by the node count during the outage. Divide by the node count.
• Circuit breaker that never closes. A breaker with no half-open probe leaves you degraded long after the store recovers — which the drill in chaos testing is designed to catch.

Verification and testing

# 1. Blackhole the store and confirm the configured behaviour on each route class.
sudo iptables -A OUTPUT -p tcp --dport 6379 -j DROP

# public read: expect 200 (fail-open) and a degraded metric
curl -s -o /dev/null -w "read:  %{http_code}  %{time_total}s\n" \
  -H "X-API-Key: $KEY" https://staging.example.com/v1/search
# payments: expect 429 with Retry-After (fail-closed)
curl -s -D- -o /dev/null -X POST -H "X-API-Key: $KEY" \
  https://staging.example.com/v1/charges | grep -iE "^HTTP/|^retry-after"

sudo iptables -D OUTPUT -p tcp --dport 6379 -j DROP

# 2. Latency must stay bounded by the command timeout, not by the outage.
#    time_total above should be ~0.03 s plus normal handling, never 5 s.

# 3. The degraded metric must be non-zero, labelled, and alertable.
curl -s localhost:9090/metrics | grep ratelimit_degraded_total
# ratelimit_degraded_total{policy="open",key_class="search"} 412
# ratelimit_degraded_total{policy="closed",key_class="charges"} 27

# 4. After recovery, enforcement must resume — not stay open.
for i in $(seq 1 300); do
  curl -s -o /dev/null -w "%{http_code}\n" -H "X-API-Key: $KEY" \
    https://staging.example.com/v1/search
done | sort | uniq -c
# expect a mix of 200 and 429; all-200 means the breaker never closed

Check four is the one that catches the subtlest failure: a limiter that degrades correctly and then never recovers looks perfectly healthy on every dashboard while enforcing nothing at all.

Frequently Asked Questions

Related

• Redis Counter Architecture — the parent guide on the store this policy protects against.
• Chaos Testing Redis Failover for Limiters — the drill that verifies the policy.
• Redis Key Design and TTL Strategy — eviction settings that can cause a silent partial outage.
• Alerting on 429 Error Rates — separating limit rejections from degraded decisions in alerting.