I run about twenty coding agents concurrently — headless workers pulling tasks from a shared queue, unattended, around the clock. I have written before about why they are cattle, not pets, and about what a closed task actually costs. This note is about the third thing you learn, the one nobody’s demo will show you: how fleets fail.
Everything below actually happened. Fifteen workers stranded by a deploy. Seven work queues wiped by a repair tool. Twenty-two repos polluted by a test run. An entire fleet’s sessions gone in one second. None of it is hypothetical, and — this is the part worth the price of admission — none of it was the model’s fault. In a year of fleet failures, the LLM barely appears in the incident log. The agent does its job. What fails is everything you stood up around it.
When you run one agent, failure means “the agent did the wrong thing.” When you run twenty, failure changes species. It moves out of the model and into the seams: the launch path, the queue, the deploy process, the host, the data layer, the test suite, your own tooling. Those seams are where this taxonomy lives.
Eight failure modes. For each one: what it looked like from the operator’s seat, what actually caused it, and the operational rule it taught. Each of these deserves — and will get — a full postmortem of its own. This is the map.
1. The fleet you launch is not the fleet you get
The symptom is silence. You launch the fleet, go do something else, and come back to find fewer workers than you started. No errors. No crash logs. No death messages. Workers that simply never existed — you count the sessions and the number is wrong.
The cause is that construction is the most expensive moment in a worker’s life, and it happens before any of your failure handling exists. A worker booting up initializes its session, loads repo state, spins up its agent harness — all CPU-hungry, all at once. Launch twenty workers simultaneously and the machine saturates precisely at the moment every worker is at its most fragile. The late ones starve during construction and die silently, because the state-machine loop that would have noticed a failure and handled it hasn’t started yet. All of my orchestration — outcome classification, retries, timeouts — protects the loop. Nothing protected the birth.
This one is insidious because the failure is invisible by construction. A worker that dies mid-task leaves a claimed bead, a session, evidence. A worker that dies being born leaves nothing. Your monitoring says the fleet is healthy because the fleet — the one that survived — is healthy. You are not operating the fleet you think you launched.
The rule: stagger launches, and treat launch itself as load. Fleet bring-up is not an instant; it is a schedulable workload with a resource profile steeper than steady-state operation. NEEDLE launches now ramp with deliberate delays between workers, and bring-up is the one phase I still watch. The general principle generalizes past agents: any system whose failure handling lives inside the main loop has an unprotected window before the loop starts. Know how wide yours is.
2. The bead that never stops failing
The queue’s contract is simple: failed tasks go back in the queue. That contract has a degenerate case, and the degenerate case runs a space heater made of tokens.
Some tasks do not fail because of bad luck. They fail because they cannot succeed — the task is too big for one context window, the acceptance criteria contradict the repo’s reality, the premise was wrong when it was written. Re-queue one of those and the fleet forms an orderly line to burn money. A worker claims it, works earnestly, fails, re-queues it. The next worker claims it. All night. Every attempt costs a full agent run — real tokens, real minutes — and an unattended fleet will pay that cost with perfect patience until you wake up and read the ledger.
The cause is an epistemological gap: the queue cannot tell a transient failure from an impossible task. Retry is a strategy only when failure is temporary. The queue does not know that. You have to teach it.
The fix has two halves. First, a circuit breaker: failure counters on every task, and after N failed attempts the task is fenced — moved to a blocked state a worker cannot claim, waiting for a human or a triage pass. Second, and less obviously: the tasks most likely to loop are the big umbrella ones, so split umbrellas into small beads and block the umbrella on its children. Small tasks fail transiently; sprawling tasks fail structurally. Task granularity turns out to be a reliability parameter, not just a planning preference.
The rule: an unattended fleet needs an explicit answer to “when do we stop trying?” — because the default answer is never, and never is expensive.
3. The deploy that stranded fifteen workers
I shipped a new orchestrator build the lazy way: moved the fresh binary over the path the fleet spawns from. Fifteen running workers detached on the spot. Still running. Still holding claims. Still spending money. No longer mine.
Here is the mechanism. Workers re-invoke the orchestrator binary through its path as they cycle. Replace the file at that path and you have not “upgraded” the running workers — you have changed the world underneath them. The processes that were mid-flight kept running the old code with nothing supervising them; a zombie herd, invisible to the new orchestrator, grazing on the same queue. The cleanup was manual: hunt down every stranded session, reconcile every orphaned claim, apologize to the ledger.
The embarrassing part is that I would never do this to a production service. Rolling deploys, drain-then-replace, immutable artifacts — I know the liturgy. But the fleet did not feel like production. It felt like my tooling, on my box, so it got the mv treatment. That instinct — the fleet is casual infrastructure — is the actual root cause, and it recurs in half the failures in this taxonomy.
The rule: deploy to the fleet the way the fleet deploys to prod. There is exactly one designated stable path that live workers reference, replacements land there atomically, and a running fleet gets stopped — drained, claims released — before the binary it re-executes changes underneath it. Never mutate the live spawn path. The fleet is production. It always was; the only question is how expensive the lesson is on the day you learn it.
4. The night the substrate died
Every session vanished at once. Twenty-some workers, the monitoring panes, everything — gone between one glance and the next. My first theory was a mass kill: something OOM-killed the workers. Wrong, and the wrongness is the lesson. The workers did not die. The thing they all lived inside died.
The fleet ran as tmux sessions, and tmux has a property that is easy to know and hard to feel: every session on the socket is hosted by one server process. That server had been quietly accumulating memory for weeks. When the kernel finally OOM-killed it, it took every session with it — not worker by worker, but all at once, one process death fanning out into total fleet loss. My orchestration had an answer for every way a worker could die. It had no concept of the layer below the workers dying, because I had never written that layer down as a dependency.
Worse, the obvious response — reboot the box — is the wrong one. It destroys the evidence and everything else that was still alive, to fix a problem one systemd unit could prevent.
The rule: inventory the processes your whole fleet transitively lives inside, and govern their resources explicitly. For me that meant the tmux server gets a systemd MemoryHigh ceiling of 48G so memory pressure throttles it long before the OOM killer arrives. But the checklist matters more than the fix: the terminal multiplexer, the SSH daemon, the container runtime, the DNS resolver — any of them is a single point of failure standing under your whole herd. You cannot circuit-break your way around a dead host process. Find your substrate before it finds you.
5. The repair that ate the queue
The work queue underneath my fleet keeps state in two places: a live SQLite database, and a JSONL checkpoint file that gets written on explicit flush. The database is the truth; the checkpoint is a snapshot of the truth as of the last flush. The queue’s repair tool rebuilds the database from the checkpoint.
You can see it coming. One afternoon, chasing corruption, I ran repair across my workspaces — without flushing first. The tool did exactly what it was designed to do: rebuilt each database from its checkpoint, discarding everything created since. Seven workspaces out of seven, every unflushed task gone. Not corrupted — cleanly, correctly deleted by a tool doing its job well.
The cause is an authority-model confusion, and it is endemic to any system with a live store and a backup: the moment you treat the checkpoint as the truth, every recovery action becomes a rollback. Nothing in the tool’s name said so. “Repair” sounds conservative. It was actually “restore,” and restore is destructive in the direction nobody thinks about — forward.
The rule comes in layers. Know, for every stateful system in the fleet, which copy is authoritative — and write it down where the 2 a.m. version of you will see it. Sequence every recovery as checkpoint first, rebuild second; in my case that is a flush before any repair, in that order, always. And push the guard into the tool itself: the upstream fix ships repair that refuses to run against unflushed state. Habits protected by tooling survive; habits protected by memory last until the afternoon you are tired.
6. The test run that lied to twenty-two repos
The dispatch pipeline needed end-to-end tests — real workers claiming real tasks and reporting real outcomes, because anything less mocks away the exact seams this taxonomy says are dangerous. The tests worked beautifully. They exercised claiming, assignment, completion, the whole loop.
They exercised it against production state. The test harness discovered workspaces the same way real workers do, and it dutifully sprayed synthetic claims across twenty-two of the twenty-nine live workspaces on the machine. Fake assignees on real tasks. Fabricated completion claims on work nobody did. And the cleanup was worse than the mess: the tooling’s own validation — sensible rules for normal operation — rejected parts of the undo, because “blank out this assignee” is not a thing the CLI believed anyone should legitimately want.
The failure is easy to caricature as a missing test fixture, but the real cause is structural. In an agent fleet, the test traffic is agent traffic — same tools, same stores, same discovery paths. There is no natural fence between “testing the fleet” and “operating the fleet” unless you build one, and nothing in the stack builds it for you. Meanwhile the blast radius of unfenced tests scales with exactly the thing you are proudest of: how much your automation can reach.
The rule: fencing test state is a fleet feature, not a test-suite nicety. Test runs get their own stores, their own workspace roots, their own discovery scope — enforced by the infrastructure, not by the test author remembering a flag. And your tooling needs an admin path to undo what automation did, because one day the thing that needs reverting will be something validation swears is impossible.
7. The claims that leak
This one is slow, which makes it the most dangerous entry in the taxonomy. Nothing crashes. Nothing pages. The fleet just gets gradually, mysteriously idle.
The mechanism: a worker claims a task, marks it in-progress, and then dies — construction kill, OOM, a harness crash, any of the deaths above. The claim survives the claimant. That task is now in limbo: not done, so the work is missing; not available, so no other worker can take it. One leaked claim is noise. But an unattended fleet manufactures dead workers at a steady rate, and each death can pin a task. Over days the ready pool quietly drains until you have a queue that looks full and a fleet with nothing it is allowed to do — starvation with a healthy-looking dashboard.
The tempting fix is a janitor: some external process that sweeps for stale in-progress tasks and releases them. The trap is that “stale” is a guess. Sweep too aggressively and you release a claim whose worker is alive and mid-task — now two workers hold one bead and you have manufactured the exact double-write the atomic claim existed to prevent.
The rule: reclamation must live inside the worker loop, governed by the same claim discipline as everything else. In NEEDLE this is the mend pass — a live worker, under the queue’s own concurrency rules, is the only thing allowed to adjudicate a stale claim and re-queue the task. Paired with a one-loop-per-bead invariant — no agent ever spans multiple tasks, no task ever has ambiguous ownership — leases stay decidable. Dead workers are normal. Immortal claims are the bug.
8. Friendly fire from the janitor
The most dangerous command in my fleet was the one that sounded safest. cleanup — the housekeeping tool that tidies up dead worker sessions. Run it casually, as the name invites you to, and it would SIGHUP live workers mid-task. The tool could not reliably tell a dead session from a working one, and it resolved its uncertainty in the worst possible direction: kill.
The precise irony: the janitor exists because of failure mode 7 — dead workers leave debris, debris needs sweeping. The remedy for one fleet pathology became a pathology. And unlike the others in this taxonomy, this one has a human trigger. Nothing about the fleet has to be wrong; the operator just has to run the reassuringly-named command at the wrong moment. I now treat it the way I treat --force flags: a thing whose invocation is itself an incident risk.
Two rules, one technical and one linguistic. Technical: any tool that acts on sessions must prove liveness before acting — a positive signal that this session is dead, not an absence-of-evidence heuristic. In fleet terms, the janitor needs the same discipline as the mend pass: destructive action only on affirmatively-verified corpses. Linguistic: name destructive tools like what they are. “Cleanup” invites casual use; a name that says this kills sessions buys you the two seconds of hesitation that would have saved my afternoon. Your fleet’s command-line surface is part of its safety model, because the operator is part of the system — and at 2 a.m., the operator is the least reliable component in it.
The pattern behind the pattern
Read the taxonomy again and notice who is missing: the model. Eight fleet-killing failure modes and not one of them is “the LLM wrote bad code.” The agent — the part everyone debates, benchmarks, and blogs about — is the component that held up.
Every failure lives at a seam. Fleet↔host: the construction kill, the substrate OOM. Fleet↔queue: the death loop, the leaked claims. Fleet↔deploy: the stranded workers. Fleet↔data: the repair wipe. Fleet↔test: the pollution. Fleet↔operator: the janitor. And every fix is almost embarrassingly boring — stagger the launches, count the failures, fence the state, order the flush before the rebuild, cap the memory, lease the claims, verify liveness before killing. No prompt engineering. No model selection. Ops discipline, applied one level up the stack than we are used to applying it.
This is the shape of the whole discipline: the agent is the easy part. We spent fifteen years learning that running software reliably is a different profession from writing it — that is why SRE exists. Agent fleets are re-running that history at fast-forward. Everyone is hiring for the software engineering; the outages come from the operations.
Naming the discipline
There is agent building — prompts, tools, harnesses, evals. It has books, courses, and a job title. Then there is the thing this note is actually about: keeping a herd of agents alive, honest, and economical, unattended, for months. Launch policy, queue semantics, deploy discipline, substrate governance, state authority, test fencing, claim leases, operator safety. That thing does not have a name yet.
I have started calling it agent fleet operations, and I think it becomes a real discipline on a predictable schedule: every team currently celebrating their first autonomous agent is somewhere on the road to their own version of this taxonomy. One agent is a program. Twenty agents are a production system, with a production system’s failure surface — plus a few failure modes that are genuinely new, because the workers spend money, fabricate state when tested carelessly, and die at rates no human team does.
Each of the eight modes above gets its own full postmortem — timeline, diagnosis, the fix’s actual design — as a follow-up note. Consider this the index. If you are building toward your own fleet: I hope the taxonomy costs you less than it cost me to write.
— Jed
The fleet in this note runs on NEEDLE (Rust orchestrator: deterministic worker state machine, agent-agnostic harness adapters), with bead-forge as the shared work queue, claude-governor as the fleet-level spend gate, and ccdash for herd health. Every incident above shaped a feature in one of them.