A broken check is not the same kind of problem as a bug
A bug in a product is discovered by the product behaving badly. Something breaks, somebody notices, and the noticing is the easy part. A bug in the thing that checks the product has no such property. It produces a number, the number looks like work, and everyone downstream treats it as the ground truth it claims to be.
That makes the failure mode specific and worth naming. A check can be wrong in two directions, and they are not symmetrical. Wrong in the alarming direction is self-correcting: somebody investigates a scary number, finds nothing, and fixes the check. Wrong in the reassuring direction is not, because nobody investigates a green tick.
We publish our own defects, so over the last fortnight we ended up with an unusually well-documented set of these. Five, in fourteen days, in a codebase with a test suite that runs on every commit. They are set out below with the commit that fixed each one, because a claim like this is worth nothing without somewhere to check it.
One: a scanner that counted words as prices
We publish no prices, and a script enforces it by scanning every published passage for anything that reads as a money figure. It reported that 70 pages carried figures with no source behind them. We published that number in a research piece, held a test ceiling at it, and worked against it for five days.
The real number was ten. The pattern for rupee amounts had no word boundary in front of it, so it matched the last two letters of any ordinary word followed by punctuation. "For years," was a price. "Two developers," was a price. "Forty hours," was a price. Sixty of the seventy were English.
The damage was not the wrong number. It was that a ceiling of seventy is a number nobody can act on, so the ten pages that genuinely needed attention sat behind sixty that did not. An overstated problem hides a real one exactly as well as an understated one does.
There is a second-order lesson underneath it. The matcher was deliberately shared between the script that reports and the test that gates, on the correct reasoning that two copies of a regular expression drift apart and a drifting matcher makes a ceiling meaningless. That was right. It also meant one bug was authoritative in both places at once, and the test agreeing with the script was structurally guaranteed rather than evidence of anything. Fixed in `433e61a`.
Two: a sitemap that said nothing had changed
Search engines decide what to re-crawl largely from the modification date a sitemap advertises. Ours advertised the twenty-fifth of July on every hub page: all four of the deep-content hubs, every topic index, and seventeen of the nineteen static routes. It had been correct once and then five batches of publishing went by without anybody bumping it, because it was a single hand-maintained constant and nothing failed when it went stale.
So for six weeks, every time a crawler read that file, it was told that the pages leading to the newest content had not changed since before that content existed. We had spent that time wondering why the new pages were not being indexed.
The fix was to derive each hub's date from the newest thing it lists, which is the rule the rest of the content register already followed. The file had no test of any kind before this, which is how a four-hundred-URL artefact went six weeks wrong without anything noticing. Fixed in `b4ed80e`.
Three: a link checker that reported a live page as dead
Our citation checker carries a rule at the top of the file, in capitals, learned from a previous tool: a HEAD request may never condemn a link. Some servers answer HEAD with a 404 and then serve the page perfectly on GET, so HEAD is used as a cheap first pass and anything that is not a success gets re-checked with a real request.
The rule was implemented for a HEAD that returns a bad status. It was not implemented for a HEAD that throws. Some hosts reject the method at the connection level rather than answering it, and for those the exception escaped before the GET could run, so the link was recorded as no response. A page returning 200 to any ordinary request was reported as unreachable.
The rule was right, written down, prominent, and explained. It was still only half implemented. A documented principle is not a guarantee that the code does it, and the gap between the two is invisible precisely because the comment reads as though it has been handled. Fixed in `ca1ac91`.
Four: two tests that could not fail
The first was quiet. A test referenced a field that does not exist on the type it was reading, and passed. The test runner does not typecheck, so the property was simply undefined, the assertion compared undefined against nothing, and the suite went green. The build caught it later. That is the shape twice now: local tests pass, the compiler disagrees, and only one of those two ran before the commit.
The second was worse and is the reason this piece exists. Writing a guard to prove that a generated file contains no hostname, the word-boundary escape in its regular expression was replaced, in transit through a shell, with a literal control character. The pattern silently became one that required an invisible byte before every domain, so it could never match anything. The test passed. It would have passed forever.
It was caught only because we deliberately corrupted the data to watch the test fail, which is a standing rule here and which on that day earned its keep. The first attempt at that proof was itself broken: the corruption did not apply, the test passed, and for a minute the conclusion was that the guard worked. Proving a gate can fail requires proving that the thing you did to break it actually broke it.
The same control-character corruption had already hit this repository once before, in the price matcher. Recognising it the second time took minutes. Recognising it the first time took a while, and in between there was a regular expression in a shipped file that quietly matched nothing.
What the five have in common
- Every one failed towards reassurance. A green tick, a clean report, or a number that made the problem look handled. None of them produced an alarming result nobody could reproduce, which is the failure that fixes itself.
- Three were in the boundary between a stated rule and its implementation. The rule was written down and correct in all three cases. The code did most of it.
- Two involved a number that was published before anybody asked what its denominator was.
- None was found by review. Two were found by the compiler, one by deliberately corrupting data, one by reading a report closely enough to notice a figure was implausible, and one by an external crawl disagreeing with us.
The practical conclusion is narrow and we would rather state it narrowly than dress it up. A check is code, code has defects, and a check's defects are less visible than the ones it is looking for. So the useful habit is not writing more checks. It is making each one fail on purpose, once, in front of you, before you believe a single number it produces.
That habit is cheap and it is not free. Breaking something to watch a test catch it takes a couple of minutes per gate, and it caught two of the five above. The other three were caught by having a second, independent instrument that disagreed: a compiler, a crawler, a person reading the number and finding it implausible. Both are worth having, because they catch different things.
What this does not cover
- Five cases from one codebase over one fortnight. That is an existence proof, not a rate. We are not claiming this is how often it happens generally, because we have not measured anybody else's tooling and could not.
- The set is biased towards what we happened to look at. There is no way to count the checks that are wrong right now and have not been caught, and the honest position is that the number is not zero.
- This is about automated checks on content and infrastructure. It says nothing about test suites for application logic, which have different failure modes and a much larger literature.
- Two of the five share a single root cause, a control character introduced by a shell. Counting them separately is defensible because they surfaced in unrelated files months apart, but a reader who wants to call it four cases rather than five is not wrong.
- Every fix here is verified by a test that was watched failing first. That is a stronger claim than most, and it is still a claim about the tests we wrote, not about the ones we did not think to write.
Sources
- This repository's own history, 2026-08-20 to 2026-09-03. Each case is a defect found in our own tooling, reproduced, fixed, and covered by a test that fails without the fix. The commit that fixed each is named in the section describing it, so every claim on this page resolves to a diff. Run: whatscene.in git history: 433e61a, b4ed80e, ca1ac91, 6d8180a, and the correction in 9532892.
- whatscene.in published money figures, whole corpus. The scan whose matcher is described in case one, before and after the fix. The corrected artifact reports the figures this page cites and is regenerated rather than edited. Run: scripts/price-claims.mjs, recorded in content/PRICE-CLAIMS.md.
Revisions
- 3 September 2026 First published.
This page is revised in place rather than replaced, so its address does not change.