It started as five copies of the same twenty lines
Five public route handlers each carried their own rate limiter. Byte-identical, copied along with each new endpoint. Five copies is five places to fix a bug and five chances to fix only four of them, so the first job was to make it one module.
Consolidating it meant reading it properly, which is how the actual problem surfaced. The limiter worked exactly as written. What it was worth was another matter.
The key was the problem
A rate limiter is a counter and a key. The counter is arithmetic and hard to get wrong. The key is a claim about who the caller is, and that is where the security lives.
Ours keyed on the client IP, taken the usual way:
const xff = request.headers.get("x-forwarded-for");
return (xff ? xff.split(",")[0] : "").trim() || "unknown";X-Forwarded-For is a header. Proxies append to it, and the first entry is conventionally the original client. But it arrives in the request, which means the caller writes it, which means the caller chooses their own key. Send a different value on every request and every request is a new stranger with a full budget.
So the limiter stopped accidents and not attempts. A retry loop, a broken client, a burst from one real browser: all correctly throttled, because none of them is trying to get around anything. Anyone deliberately rotating the header was never limited at all.
The header is fine as a throttling key and useless as identity. The mistake is not using it. The mistake is expecting a security property from it.
There was a second, quieter issue. The counter lives in the instance's memory, so two Cloud Run instances keep two independent counters and the real ceiling is the configured limit times however many instances happen to be up. That number is not fixed and not something we choose.
What it was worth to an attacker
The endpoint that made this matter is the error sink. It is unauthenticated by design, because a page that has just broken needs to be able to report it, and every accepted request becomes a database write.
An unauthenticated endpoint that turns requests into writes is a billable write amplifier. The per-IP limit was the only thing standing in front of it, and the per-IP limit could be shed by changing one header. So the spend was unbounded, and bounded only per fabricated address, which is not a bound.
No availability story, no data exposure. Just a bill that could be run up by anyone who noticed, with no login and no cleverness.
The fix we deliberately did not do
The correct fix for a spoofable forwarding header is to stop reading the first entry and start reading the entry your own infrastructure wrote. That needs the trusted hop index: how many proxies sit in front of your container, counted from the right.
We did not do it, and the reason is that we could not establish the number honestly. It is a runtime property of the hosting platform, not something the code can assert, and guessing is worse than leaving it. Pick an index too shallow and you are still trusting a value the caller controls, which is where we already were. Pick one too deep and you key every visitor in the world onto one shared frontend address, throttling all of your real traffic together while an attacker is unaffected.
The other half of a proper fix is a shared counter store, so all instances count together. That is a real dependency and a round trip on every request, for an endpoint whose entire job is to be cheap.
So the per-key limit is still spoofable today. It is documented as spoofable, in the module, at the top, in the place someone changing it will look. Writing it down is not a fix and we are not presenting it as one.
The fix we did: a limit keyed on nothing
If the problem is that the key is attacker-controlled, one answer is a limit with no key at all.
export function ceiling(max: number, windowMs: number) {
let hits: number[] = [];
return function exceeded(): boolean {
const now = Date.now();
hits = hits.filter((t) => now - t < windowMs);
hits.push(now);
return hits.length > max;
};
}It counts every accepted call on the instance and ignores the caller entirely. Nothing in a request can raise it, rotate it, or route around it, because it never looks at the request. That is the whole property, and it is achieved by removing code rather than adding it.
This converts the risk from unbounded spend to bounded spend. It does not stop a flood and does not identify anyone. It puts a number on the worst case, which for a write amplifier is the thing that actually needed a number.
The trade is real and worth naming: under a flood the ceiling drops legitimate error reports too. For an error sink that is the right way round. Losing visibility for a few minutes is recoverable; an unbounded bill is not.
One more thing came out of the rewrite. A rotating key is also a slow memory leak, since every fabricated value creates a map entry. The same attack pointed at a different resource. The per-key limiter now prunes expired entries once it passes 5,000 keys.
The test asserts the vulnerability
The regression test for this is unusual, because the useful thing to pin is not that the limiter works. It is that the limiter is still defeatable and the ceiling still is not.
So the assertion named "cannot be influenced by anything in a request" checks that a rotating key does escape the per-key limit, and that the ceiling holds anyway. If somebody later fixes the trusted-hop problem, that test will fail, and it should: the situation it describes will have changed.
A test that encodes a known weakness is worth more than a comment describing it. The comment goes stale silently. The test fails loudly the day the weakness stops being true.
What to take from it
- Ask what your rate limiter is keyed on, and then ask who writes that value. If the answer is the caller, you are throttling accidents, not attempts. That may be all you need, but decide it rather than inherit it.
- An unauthenticated endpoint that costs money per request needs a limit that no request can influence. Keyed on nothing is a legitimate design, not a fallback.
- In-memory counters multiply by instance count. That number is elastic, so your effective ceiling is elastic too, in the direction you did not want.
- A rotating key is a memory leak as well as a bypass. Bound the map.
- When you cannot fix something properly, write down precisely why, next to the code. Then write a test that will fail when the reason stops applying.
We are publishing this one specifically because it is unresolved. The tidy version of this story would stop after the ceiling. The useful version says which threat is handled, which is not, and what it would take, because the shape of the compromise is more transferable than the fix.
What this does not cover
- The per-key limit described here is still spoofable in production. This piece describes a mitigation and a documented gap, not a resolved vulnerability.
- The ceiling is per instance, so the true system-wide bound is the configured value times the number of running instances. We do not control that number and do not claim a fixed figure.
- We have not measured an actual flood against our own endpoint. The reasoning about cost is arithmetic about writes per request, not an observed incident.
- Trusted-hop configuration is specific to a hosting platform. Anyone copying this should establish their own proxy depth rather than adopting a number from someone else's setup.
- This covers request throttling only. It says nothing about the other reasons an endpoint might need protecting, such as input validation or authentication.
Sources
- lib/rate-limit.ts and tests/rate-limit.test.ts in the whatscene.in repository. The module, its documented reasoning, and the assertion that pins both the remaining weakness and the ceiling that compensates for it. Run: commit b995675, 2026-08-20.
- X-Forwarded-For, MDN Web Docs. Retrieved 24 August 2026.
Revisions
- 24 August 2026 First published. The ceiling was added on 20 August 2026.
This page is revised in place rather than replaced, so its address does not change.