A mute gate is worse than a red one

The dashboard said nothing was wrong, and it was right in the most useless way possible. My fleet’s CI hadn’t failed a single build all day. It also hadn’t passed one. It had started twenty-five and finished none.

Every gate you build is designed around two states. Green means go; red means stop and look. Both are verdicts — a claim about a specific commit, which somebody can act on. There’s a third state nobody designs for, because it doesn’t look like a state at all: the gate that says nothing. No red, no green, no error. Just a run list full of builds that were started and never concluded.

I’ve started calling this a mute gate, and I’ve come to believe it’s strictly worse than a red one. A red gate is a problem you’ve been told about. A mute gate is a problem that looks exactly like “fine.”

Twenty-five builds, zero verdicts#

The setup was ordinary. A build on that repo takes 18 to 29 minutes. The fleet — twenty-odd headless coding agents pushing to the same main — lands a commit every 2 to 8 minutes during working hours.

A week earlier I’d fixed a real problem: stale builds. Six of them had burned between 30 and 82 minutes each testing commits that were already obsolete by the time they started. So I added the most standard setting in CI, the one GitHub Actions exposes as cancel-in-progress: true: when a newer commit arrives, kill the build that’s running for the older one. Test the latest code, not history. Every CI system offers some version of it, and every one I’ve used makes it look like pure hygiene.

Do the arithmetic I didn’t do. If a build takes 24 minutes and a push arrives every 5, a build only finishes if the fleet happens to go quiet for 24 straight minutes. During working hours, it never did. Each push killed the build in flight, started a new one, and that one was killed by the next push. Twenty-five builds, every one of them superseded a third of the way through.

The part that should worry you is how normal this looked from the outside. A superseded build isn’t a failed build. It shows up as stopped — in my case, exit code 143, which is just SIGTERM, the process being told to go away. Nothing turned red. Nothing paged. The run list was a column of builds that started, ran a while, and quietly ended, which is also what a healthy pipeline looks like when you glance at it between other things.

It doesn’t degrade. It falls off a cliff.#

The intuition that got me here was that cancelling stale builds costs a little throughput and saves a lot of waste — a gentle trade. The math says otherwise, and the shape of it is worth seeing.

Say pushes arrive at random, averaging one every I minutes, and a build takes D. Under cancel-in-progress, a build finishes only if no push lands during its entire run. For random arrivals the chance of that is e^(−D/I). At my numbers — D = 24, I ≈ 5 — that’s e^(−4.8): 0.8% of builds finish, about one verdict every ten hours. In practice it was worse, because pushes come in bursts during the day: none, all day.

Now change one word in the policy. Cancel pending instead of in-progress: a build that has started always finishes; a newer push replaces a queued build that hasn’t started yet, which costs nothing because no work was done. The gate now produces one verdict per build, back to back, whenever there’s anything to test: at my numbers, about one every 24 minutes. Same pushes, same builds, same cluster. The difference is one policy word, and it’s about twenty-five times more verdicts.

The cancel-in-progress curve peaks where pushes arrive exactly as often as builds take, and below that it doesn’t decline gracefully — it collapses exponentially. Anything that raises your push rate walks you down that curve: more agents, more active hours, smaller commits (which is otherwise good advice). The dangerous property is that nothing about the gate itself tells you which side of the peak you’re on. It just goes quiet.

Why mute is worse than red#

A red gate is a message. It names a commit, a test, a line number. It’s annoying, and it’s useful: somebody looks, something gets fixed, and in the meantime everyone knows not to trust main.

A mute gate sends no message, and the absence of a message reads as good news. Nobody investigates silence. And an unattended fleet reads silence the same way a human does, only faster. During the stretch that CI was muted, 575 commits landed with the definition-of-done check bypassed, and no release was cut for twelve days. Every individual bypass was locally reasonable — the gate wasn’t going to say anything either way, so why wait on it? Twelve days of reasonable decisions add up to a codebase nobody has actually verified.

There’s a second way to go mute, and it starts red. A few weeks earlier the same CI had been red for 36 hours because one test deadlocked and timed out on every run. A red gate that stays red stops carrying information too: it can’t distinguish “my commit is fine” from “I just broke something,” so it gets ignored for the same reason a mute one does. About 225 commits landed with the pre-commit check skipped. When the deadlock was fixed, 78 tests had drifted underneath it, and within the hour main stopped compiling outright. A gate that’s always red is mute with extra steps.

The flaky gate: failing work that’s already done#

The third variety is the most insidious, because it does produce verdicts. They just aren’t about your code.

My fleet runs a definition-of-done gate after every task: the agent does the work, and the gate decides whether it counts. On one repo, on one day, 18 of 66 attempts were released as gate failures — about 27% of that repo’s spend. One task in particular failed eight times over three rounds and was automatically quarantined. Thirty-six of the repo’s 41 open tasks depended on it. The ready queue emptied: the only things left were the quarantined task and a few waiting on humans. A fleet with literally nothing to claim.

The task’s work had been finished four times over. The code was committed on main. Its acceptance tests passed. All fifteen subtasks it had been split into were closed. Every one of those eight failures was the gate, not the work — two flaky tests in an unrelated part of the suite:

• A small test HTTP server inherited its standard library’s default listen backlog of 5, while its own contract test opened 32 simultaneous connections. When the accept queue overflowed, clients saw connection resets. Sometimes.
• A test asserted on a log line that the handler writes after sending the response. The client usually got there second. Sometimes it got there first, and the count read 0 instead of 1.

A flaky gate is a mute gate wearing a red costume. Its verdict carries information about timing and luck, not about the commit under test. And it conceals itself in a way the other two don’t: during the incident, a “fix the build” task was raised and then closed as no change needed, main already passes — because the flake happened to pass on the run that checked. One green run is not evidence about a flake. It’s a coin that happened to land heads.

What I run now#

None of these fixes is clever. They all follow from one idea: a gate’s product is a verdict, so measure the verdicts, not the colours.

Alert on silence, not just on red. The most important number about a gate isn’t its pass rate — it’s the time since it last produced a verdict under its own power. A cancelled build isn’t evidence the gate can finish anything, so it doesn’t reset the clock. When that clock passes a threshold, that’s an incident, even though nothing is red.

Cancel pending, never in-progress. Queued builds can be superseded freely; a started build finishes. And as a belt with those braces, a starvation guard: if nothing has finished under its own power in the last 45 minutes, supersede nothing on this pass. If there’s no evidence the gate can complete, don’t assume it can.

Red is a stop-the-line event. When a gate goes red and stays red, restoring green is the first task, not a side quest, and pushes to that repo pause until it’s done. Fixing drifted tests while the fleet keeps pushing on top of them is bailing out a boat with the tap still running.

Treat flake rate as a throughput tax — and prove fixes by repetition. Don’t diagnose a flake by re-running until it’s green. Force the race: shrink the queue, inject the delay in the late writer, and show that the old code fails every time and the new code passes. Then prove it with a streak — I use fifteen consecutive green runs of the full gate, and I report the count. One green proves nothing about a flake, which is exactly how the flake survived the first time.

When finished work keeps failing, suspect the gate first. A task that cycles with “verified success” from the agent and “gate failed” from the harness is telling you where the problem is. Read the attempt history before you re-dispatch, split, or rewrite the task. Otherwise you spend another round rediscovering that the work was done.

Same failure, one laptop#

None of this needs a fleet. You’ve probably built a mute gate at your own desk.

A test watcher that restarts on every save is cancel-in-progress with a file watcher. If your suite takes 40 seconds and you save every 15, it never finishes — and you’ll read the absence of red output as “tests are passing.” The same goes for an agent session: ask an agent to run a long suite with a short timeout and it will come back saying the tests “are running” or “look good so far.” That’s not a verdict. It’s the agent grading its own homework on a test that never finished.

Scale changes the cost, not the shape. On a laptop a mute gate wastes an afternoon. Across twenty unattended workers it wastes twelve days and five hundred commits — because a fleet doesn’t wait for a verdict that isn’t coming. It builds on top of the silence.

The question I ask now#

When a gate hasn’t bothered me in a while, I no longer take that as good news. I ask:

When did this gate last actually say something?

If the answer is “twenty minutes ago, it passed,” good — that’s a gate. If the answer is “I’m not sure,” or “it’s been red since Tuesday,” or “it passed once, on the retry,” then what I have isn’t a gate. It’s a decoration on the pipeline, and everything that has merged past it since then is unverified.

Red is a problem you’ve been told about. Build your gates so that silence is one too.

— Jed

Related: Don’t let the agent grade its own homework — why “done” needs an independent verdict in the first place. And Trust is a property of the system, not the agent — the same argument about safety: build the check into the system instead of relying on someone to notice.