Teaching the Factory to Pick Its Own Tools

A few weeks ago I was auditing a production task in Composer — my orchestration platform for AI coding agents — and found a run that claimed it cost $54.14.

That number was wrong. The task had run on GLM-5.2, an open-weight model served through Ollama's cloud, but the CLI had priced the tokens at Opus rates. The real cost was about $5.79. A tenth of the sticker price.

Fixing that turned into a five-phase cost-accounting overhaul — write-time price snapshots, per-model trust flags, the works. But the bug left me with a more uncomfortable observation than any accounting error.

If the same task costs $5.79 on one model and $54 on another, then the single biggest cost decision in my entire platform is which model picks up the ticket.

I was making that decision with a regex.

Auto Mode Was a Regex

Composer has had an "auto" model mode for a while. It works — in the way a thing works when better machinery downstream absorbs its mistakes.

Here's the whole algorithm, honestly stated. Match the prompt against a handful of keyword patterns: auth, migration, concurrency, refactor mean the task is large; typo, docs, rename, lint mean small. Count the characters — over 1,800 means large, under 450 means small. Everything else is medium. Each size maps to a fixed ladder of models, and the first one with quota headroom gets the job.

No memory of outcomes. No idea what each model is actually good at. A task classified as a feature could never be small, even if it was a one-line copy change, because the type check short-circuited before the keyword check ever ran.

Worst of all: nothing ever escalated. When a code review came back REVISE, or a task ended blocked with no pull request, the retry ran on the exact same model that just failed. No human lead staffs work that way. The second attempt goes to someone more senior — that's practically the definition of escalation.

The review loops and quota gates I wrote about in The Inverse Architecture were quietly paying for all of this. Quality stayed high because the harness caught the misroutes. It just caught them expensively.

Then a Paper Landed

The nudge to finally fix it came from an arXiv paper: Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving.

The idea is a two-stage cascade. Stage 1 routes each incoming query to the cheapest model that can handle it, using clusters of similar queries and each model's measured error rate per cluster — with a single interpretable knob that trades accuracy against cost. Stage 2 inspects the cheap model's output with a lightweight classifier and escalates anything that looks low-quality to a stronger model.

The result that got my attention: the cascaded system kept 97–99% of the strongest model's accuracy while cutting serving cost meaningfully. You don't pay frontier prices for easy queries, and a safety net catches the hard ones that slipped through.

Halfway through the paper I realized Composer already had almost every ingredient. Ground-truth outcomes? My pollers verify every pull request against GitHub — merged or not, review verdicts per round, failure reasons, token spend. An escalation trigger? The code reviewer already returns APPROVED or REVISE on every PR. The paper's missing piece in my platform was embarrassingly specific: nothing connected those signals back to the routing decision.

The paper's setting doesn't transfer directly, to be fair. It clusters 9,000 short queries with k-means; I dispatch tens of long-horizon coding tasks per week, and my "correctness label" arrives hours later when a PR merges. The architecture transfers. The machinery has to be rebuilt for my scale.

Do the Cheap Models Deserve the Work?

Routing cheap only makes sense if the cheap models can actually carry real work. So before designing anything, I made Claude research the current state of every model in my pool — the Anthropic lineup plus the open-weight models Ollama serves — against agentic coding benchmarks specifically. Not chat trivia. Repo-scale, tool-using, PR-writing benchmarks like SWE-bench Pro and Terminal-Bench.

The headline finding reshaped the whole design:

Sit with that GLM-5.2 row for a second. An open-weight model, served on a flat-rate cloud plan, scoring within a point of Sonnet 5 on the benchmark that best approximates my actual workload — real GitHub issues resolved end to end. Two years ago "open model" meant a toy for chat demos. The middle class is real now.

The research cut the other way too. Two models in my pool failed the "would I let this thing write a PR unsupervised?" test outright. They stay in the pool for cheap utility calls — but the router needs to know they can never be the lead agent on a task. Capability knowledge that only exists as vibes in my head is useless to a dispatch loop; it has to be data.

Route Cheap, Escalate on Evidence

Here's the design that came out of the planning session, mapped stage-for-stage onto the paper.

Assess. The paper clusters queries by embedding similarity. At my volume, that's replaced by scored heuristic signals — risk keywords, scope language, size bands, boundedness — that produce a complexity tier plus a confidence flag. High-confidence cases route immediately, for free. Low-confidence cases — the ambiguous middle — get one extra opinion: a single stateless Claude Haiku call that reads the task against a capability cheat sheet and returns a tier. Cheap gates protect expensive models. That principle carried the last post; it picks the triage model here.

Route. Each tier walks a ladder of models, cheapest-capable first, taking the first one with quota headroom. The interesting ladder is the medium tier, because it's quota-adaptive: while my Anthropic 7-day window sits under 50%, medium tasks default to Opus 5 — capability is cheap when the budget is fresh. Past 50%, the ladder flips and GLM-5.2 takes the lead, with Sonnet as fallback. The λ knob from the paper, translated into the currency that actually matters on a subscription plan: quota survival.

Escalate. This is the paper's Stage 2, and the piece I'm most glad to finally have. When a review comes back REVISE a second time, or a task dies blocked with no PR, the retry doesn't rerun the same model — it walks an escalation ladder. Sonnet escalates to GLM (a deliberate cross-family second opinion at similar capability). GLM escalates to Opus — always, even under quota pressure; a rescue is exactly the wrong moment to pinch pennies, so that rung defers for the quota window rather than routing around it. Opus escalates to Fable. Two escalations max, and the original model keeps attribution for the stats, so the scoreboard stays honest.

Learn. The paper's error table is computed offline from training data. Mine will be computed from production: a per-model rollup — merge rate, revision rate, failure taxonomy, cost per merged PR — built from telemetry the platform already records. No automated decision reads it yet. It exists so that I can see, in a month, whether GLM's merge rate on medium tasks actually holds up, and reorder the ladders from evidence instead of benchmarks.

Every decision the router makes writes a human-readable reason into the task timeline: "7d window 62% ≥ 50% — medium walks pressured ladder." If I can't reconstruct why a model got picked, the router is broken. That's non-negotiable.

What I Deliberately Didn't Build

The restraint list matters as much as the feature list.

No k-means. No embedding pipeline. No fine-tuned quality classifier — the paper trains a ModernBERT model on thousands of labeled outputs, and I have tens of tasks a week; my "quality classifier" is the deterministic review verdict I already trust to gate merges. No bandit algorithms, no online learning. The feedback loop is a table a human reads.

This is the same inverse-architecture discipline as always: deterministic code makes every decision; AI gets hired for exactly one narrow judgment call (the ambiguous-middle triage), and even that one degrades gracefully to heuristics if the call fails. Each new behavior ships behind a setting, so rolling back any piece of this is a toggle, not a deploy.

Five PRs, Queued to the Factory

Here's the part I still find slightly absurd, in the best way.

The whole design became five phased GitHub issues — capability metadata, the quota-adaptive policy, the outcome rollup, the escalation ladder, the LLM triage. Those issues went into Composer as a chained sequence of tasks, each one gated on the previous phase merging. Opus 5 is implementing all five.

The factory is building its own router. I planned it over coffee; the agents are doing the typing; the review loops that caught the old router's mistakes will be reviewing the new router's code.

Since March, Composer has dispatched 1,102 tasks and merged 927 pull requests — a 98.3% merge rate on opened PRs, at about $6.84 per merged PR. Those are the numbers the router has to defend. Merge rate stays put, or the change rolls back. Cost per merged PR should drift down as GLM earns a bigger share of the queue. The escalation ladder's rescue rate — escalated tasks that go on to merge — tells me whether the safety net actually catches what the cheap models drop.

I'll report back with the before-and-after once the data is real. That's the difference between this post and a benchmark blog: the router doesn't have to win an argument, it has to survive production.

You wouldn't staff a typo fix with your principal engineer. You wouldn't send an intern alone into an auth refactor — but if you did, and it went sideways, you'd know exactly who to hand it to next. Routing isn't a machine-learning problem at my scale. It's a management problem, and the manager finally has a staffing policy.

–Jeremy