Verifying Rate Limits in CI Pipelines

Limits drift in three directions at once: the number in the code, the number in the gateway configuration, and the number in the public documentation. A pipeline that checks all three plus the response contract catches nearly every regression before it reaches a customer, and it turns the test layers described in rate limit testing and validation into a merge gate.

The problem in concrete numbers

A team raises the enterprise plan from 500 to 1,000 requests per second. The change lands in the application configuration. The gateway plugin override still says 500, so the gateway rejects at half the new limit. The API reference still says 500 too, so support tells the customer the old number. Three sources of truth, one change, two of them stale — and nothing failed, because no test compares them.

The check that would have caught it runs in under a second: parse all three, assert equality.

Three sources of truth for one published limit The application configuration, the gateway plugin configuration, and the public API reference each carry the same number, and a pipeline check compares all three so a change to one fails the build until the others follow. One number, three places it must match app config plans.yaml enterprise: 1000/s gateway config kong.yaml second: 500 (stale) public reference openapi.yaml 500 per second (stale) CI check: assert all three are equal runs in under a second, fails the build until they agree

What belongs at each pipeline stage

Stage Checks Runtime Blocking?
Pre-commit / unit Decision arithmetic, middleware headers < 1 s Yes
Pull request Config-versus-docs drift, integration against ephemeral Redis < 60 s Yes
Post-merge to staging Contract assertions against a deployed API 1–2 min Yes
Nightly Full load test measuring the accepted ceiling 5–10 min Alert, not block
Weekly Failure drill: store blackhole, failover 10 min Alert, not block

The split matters: a pull request cannot wait ten minutes for a load test, but it can absolutely wait sixty seconds for a drift check and an atomicity test against a container.

Step-by-step: wiring the gates

yaml
# .github/workflows/ratelimit.yml — the blocking half of the pipeline.
name: rate limits
on: [pull_request]

jobs:
  unit-and-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      # Fast, pure tests: arithmetic and middleware behaviour.
      - run: npx vitest run test/limiter test/middleware --reporter=basic
      # The drift check: one number, three artifacts.
      - run: node scripts/check-limit-drift.mjs

  integration:
    runs-on: ubuntu-latest
    services:
      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping" --health-interval 5s --health-timeout 3s --health-retries 5
        ports: ["6379:6379"]
    env:
      REDIS_URL: redis://localhost:6379
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      # Atomicity, TTLs, script loading — the things a mock cannot prove.
      - run: npx vitest run test/integration --reporter=basic
javascript
// scripts/check-limit-drift.mjs — parse all three artifacts and compare.
import { readFileSync } from "node:fs";
import YAML from "yaml";

const plans   = YAML.parse(readFileSync("config/plans.yaml", "utf8"));
const gateway = YAML.parse(readFileSync("deploy/kong.yaml", "utf8"));
const openapi = YAML.parse(readFileSync("docs/openapi.yaml", "utf8"));

const failures = [];

for (const [name, plan] of Object.entries(plans.plans)) {
  // 1. Gateway: find the plugin instance scoped to this plan's consumer group.
  const plugin = (gateway.plugins ?? []).find(
    (p) => p.name === "rate-limiting" && p.consumer_group === name);
  if (!plugin) { failures.push(`${name}: no gateway rate-limiting plugin`); continue; }
  if (plugin.config.second !== plan.perSecond) {
    failures.push(`${name}: gateway ${plugin.config.second}/s != app ${plan.perSecond}/s`);
  }

  // 2. Public reference: the documented number lives in a structured extension,
  //    not in prose, precisely so it can be diffed.
  const documented = openapi["x-rate-limits"]?.[name]?.requestsPerSecond;
  if (documented !== plan.perSecond) {
    failures.push(`${name}: docs ${documented}/s != app ${plan.perSecond}/s`);
  }

  // 3. Sanity: burst must be published too, and must not exceed the window budget.
  if (plan.burst == null) failures.push(`${name}: burst is not defined`);
}

if (failures.length) {
  console.error("rate limit drift detected:\n  " + failures.join("\n  "));
  process.exit(1);
}
console.log(`rate limits consistent across ${Object.keys(plans.plans).length} plans`);
Which limiter checks block a merge and which only alert Unit tests, drift checks, and integration tests block a pull request because they finish in under a minute; staging contract assertions block the deploy; the nightly load test and weekly failure drill raise alerts instead of blocking. Block what is fast, alert on what is slow blocking: under 60 seconds unit arithmetic, header behaviour, config drift, atomicity on a container blocking the deploy: 1 to 2 minutes live contract assertions on staging, tier ordering invariant scheduled: alerts, not gates nightly load test records the measured ceiling; weekly drill blackholes the store a ten-minute job in the merge path gets disabled the first busy week

Contract assertions against staging

Three requests are enough to catch most header regressions, and they run after every deploy.

bash
#!/usr/bin/env bash
# scripts/assert-contract.sh — run after deploying to staging.
set -euo pipefail
BASE="${1:-https://staging.example.com}"
KEY="${CI_TEST_KEY:?set CI_TEST_KEY}"
fail() { echo "CONTRACT FAIL: $*" >&2; exit 1; }

# 1. A success carries the full triplet.
hdrs=$(curl -fsS -D- -o /dev/null -H "X-API-Key: $KEY" "$BASE/v1/ping")
grep -qi '^x-ratelimit-limit:'     <<<"$hdrs" || fail "missing X-RateLimit-Limit"
grep -qi '^x-ratelimit-remaining:' <<<"$hdrs" || fail "missing X-RateLimit-Remaining"
grep -qi '^x-ratelimit-reset:'     <<<"$hdrs" || fail "missing X-RateLimit-Reset"

# 2. Drive the key over its limit and inspect the rejection.
for _ in $(seq 1 400); do curl -s -o /dev/null -H "X-API-Key: $KEY" "$BASE/v1/ping" & done; wait
rej=$(curl -s -D- -o /dev/null -H "X-API-Key: $KEY" "$BASE/v1/ping")
grep -q '^HTTP/[0-9.]* 429' <<<"$rej" || fail "expected a 429 after exceeding the limit"
ra=$(grep -i '^retry-after:' <<<"$rej" | tr -dc '0-9')
[ "${ra:-0}" -ge 1 ] || fail "Retry-After must be >= 1 (got '${ra:-none}')"

# 3. The tier invariant: a rejection must come from the application, not the edge.
grep -qi '^x-ratelimit-remaining: 0' <<<"$rej" || fail "429 lacks app headers — edge tier is too tight"

echo "rate limit contract OK against $BASE"
Three live assertions and the regressions they catch Checking quota headers on a success catches a middleware that stopped setting them, checking Retry-After on a rejection catches a zero or missing wait, and checking that a rejection carries application headers catches an edge shield that has become tighter than the published limit. Three requests, three classes of regression success has the triplet catches: middleware reordered or skipped 429 has Retry-After catches: zero or missing wait after a refactor 429 has app headers catches: edge shield tightened below the limit ten seconds of runtime covering the regressions customers notice first

Gotchas and edge cases

  • Tests that mutate a real customer’s counters. Always use a dedicated identity, and label it so alerting excludes it.
  • Parallel CI jobs sharing one Redis. Two pipelines on the same store produce phantom failures. Prefix every key with the run identifier, or use a service container per job.
  • Asserting on wall-clock durations in CI. A loaded runner turns a 100 ms expectation into 400 ms. Assert on counts and reported values instead.
  • Drift checks that parse prose. A limit written as “1,000 requests/second” in a Markdown paragraph is not machine-comparable. Keep the number in a structured field and render prose from it.
  • Staging without the production node count. The contract assertions still work, but any accepted-rate assertion is meaningless. Keep ceiling measurement in the nightly job against a realistic environment.
  • A nightly job nobody reads. Publish the measured ceiling as a build artifact and alert on change, or it becomes a green tick that means nothing.
  • Blocking merges on a flaky live check. If staging is occasionally unavailable, retry once and then skip with a warning rather than blocking every developer.

Verification and testing

bash
# Prove the gate actually catches drift: change one number and expect a failure.
sed -i 's/perSecond: 1000/perSecond: 1200/' config/plans.yaml
node scripts/check-limit-drift.mjs; echo "exit=$?"
# rate limit drift detected:
#   enterprise: gateway 1000/s != app 1200/s
#   enterprise: docs 1000/s != app 1200/s
# exit=1
git checkout config/plans.yaml

# Prove the contract script catches a missing header (simulate by pointing at a route
# that skips the limiter middleware).
./scripts/assert-contract.sh https://staging.example.com/internal || echo "caught as expected"

A gate that has never failed is a gate nobody has verified. Break each check deliberately once, confirm the pipeline goes red, and keep that exercise in the runbook for the next person who inherits the pipeline.

Frequently Asked Questions

Should a load test block pull requests?

No. A meaningful load test needs minutes and a realistic environment, which is far too slow for a merge gate and will be disabled the first time it delays a release. Run it nightly against staging or a production canary, publish the measured ceiling, and alert when it drifts.

How do I keep documentation in sync with configuration?

Generate one from the other. Where that is impractical, keep the published numbers in a structured field of your API description rather than in prose, and add a check that parses both and fails on mismatch. Numbers in sentences cannot be diffed.

Is an ephemeral Redis in CI representative enough?

For atomicity, key expiry, and script behaviour, yes — those are properties of the server, not the topology. For failover, cross-slot placement, and replication lag you need a cluster, which belongs in a scheduled drill rather than in every pull request.

What should the pipeline do when staging is down?

Retry once, then skip the live assertions with a visible warning rather than blocking merges. The blocking value is in the fast checks; the live ones are a safety net whose absence should be noticed but should not stop the team from shipping.