The Inverse Architecture: Building the System That Builds the Systems

Four months ago I wrote about building Composer — my work queue and orchestration platform for AI coding agents. I described it as a small software company where every employee is an AI and I'm the manager handing out assignments.

That post ended with a confession: my workflow would probably look completely different in a few months.

It does. But not in the way I expected. The platform didn't get replaced — it got deeper. What changed most is my job description.

I don't read the code anymore.

Not as a stunt. Not as a philosophical statement. It's just no longer where my time goes. I don't review pull requests. I don't do first-level QA. My hands-on hours go into designing workflows, tuning agent definitions, and building out the platform itself. When software ships, I'm the UAT tester and the release approval gate — the last human in the loop, standing at the very end of an assembly line that runs without me.

Since that last post, Composer has processed 793 tasks, opened 652 pull requests, and merged 642 of them — at a 94% success rate, for a total estimated cost of about $4,000. That's roughly $6.30 per merged pull request.

I want to walk you through how, because the architecture underneath those numbers is the opposite of what most of the industry is chasing right now. That inversion is the entire point of this post.

The Inverse Architecture

Here's the pattern I keep seeing everywhere: put a powerful LLM at the center, give it tools and autonomy, then spend months wrapping guardrails around it to make it behave. The agent is the brain; the software is the seatbelt.

Composer is built inside out from that.

At Composer's core sits deterministic software — plain, boring, testable TypeScript. A dispatch loop that wakes up every two seconds and pulls the highest-priority task off the queue. Pollers that watch pull requests, merges, and retrospectives on fixed intervals. A quota manager that tracks usage windows and throttles concurrency. A state machine in SQLite that owns every task transition.

None of that is AI. None of it needs to be AI. Scheduling, routing, budget enforcement, retry logic, crash recovery — these are solved problems. We've known how to build reliable queues and workflows for decades. Handing them to a stochastic model doesn't make them smarter; it makes them unpredictable.

AI gets invoked at exactly seven points in the platform — each one narrow, bounded, and gated. The implementer session that writes the code. The code reviewer that returns a verdict. The judge that evaluates acceptance criteria. The retrospective agent that looks back at merged work. A few others we'll get to.

The rule that makes it all hold together is one sentence, repeated throughout Composer's codebase and docs:

The server, not the agent, applies all state.

An agent never merges a pull request — it returns APPROVED or REVISE, and a deterministic poller does the merging. An agent never marks its own task done — completion resolves from pull request ground truth, not from the model's say-so. A prompt-injected issue body can't drive a merge, close a ticket, or touch a label, because no agent holds those keys. AI proposes. Deterministic code disposes.

There's a name for this discipline now. The term making the rounds this year is harness engineering — the craft of building the wrapper that makes agents reliable: the orchestration, the verification loops, the context management, the guardrails. The working equation is Agent = Model + Harness, and the harness is where reliability actually lives.

Anthropic's "Building Effective Agents" draws a distinction I've come to lean on: workflows are systems where LLMs are orchestrated through predefined code paths, while agents dynamically direct their own process. By that taxonomy, Composer is a workflow engine that employs agents — the paths are mine, the labor is theirs. Deterministic where determinism wins, stochastic only where judgment is genuinely required.

The models get all the headlines. The harness is where the reliability comes from. I'd rather own the harness.

Meeting the Input Where It Is

Real work doesn't arrive uniformly specified. Some days I hand Composer a fully-formed GitHub issue with phased requirements and acceptance criteria. Other days I have a fuzzy idea I can barely articulate — something about surfacing quota pressure earlier, maybe?

Early on, the loose ideas were the problem. Feed an underspecified prompt to an autonomous agent and you get confident garbage — a plausible implementation of a thing you didn't quite want. The fix wasn't better prompting. It was making the system evaluate the input and adapt the workflow to match.

Well-defined work takes the direct path. A feature task runs a single autonomous agent through the full lifecycle — design a blueprint, implement, test, document, deliver a pull request. A bugfix skips the design ceremony and goes straight to diagnose-and-fix. No meetings, no hand-offs, no waiting.

Loosely-defined work gets refined before it gets built. I jot the fuzzy thought into Composer's Ideas backlog. When I'm ready, I open a Conversation on it — a read-only chat session where Claude explores the codebase and interrogates the idea with me. No code gets written; the entire job is turning vague intent into shared understanding. When the conversation converges, one click captures it as a specification task, which produces phased GitHub issues — an implementation plan, not an implementation. Those issues then flow back through the direct path as well-defined work.

This is spec-driven development with the spec-writing itself automated. The system doesn't demand perfect tickets. It notices what it's been given and routes accordingly — refine the loose stuff, button up the tight stuff.

The part I underestimated: the Conversation step is where my thinking happens now. The refinement dialogue is doing the job that design documents and whiteboard sessions used to do — except it has read the entire codebase, and it's available at 11 PM when the idea shows up.

Quality Is a Loop, Not a Gate

The original Composer had review stages. What it has now are review cycles — loops with teeth, bounded budgets, and a strict rule about who pays for what.

Take code review. When an implementer opens a pull request, a poller picks it up on a 30-second cadence. Before any AI reviewer runs, three deterministic guards must all pass: CI has finished, no reviewers are still pending, and the merge state is clean. Only then does the code-reviewer agent get spawned — and all it can return is a verdict. APPROVED, and the poller squash-merges. REVISE, and the task re-queues with the feedback attached, the implementer addresses every comment, and the cycle runs again — up to a configurable maximum number of rounds.

The same pattern gates every phase of the newer Workflow Engine, where I define workflows as graphs of phases, each with acceptance criteria. Evaluation is two-tier and deliberately cheap-first: deterministic checks — does the file exist, does the output match this pattern, is the error sentinel absent — run first and short-circuit on the first failure with no model call at all. Only when every free check passes does an LLM-as-judge evaluate the subjective criteria — and that judge is Haiku, the cheapest model in the lineup, because "did this output meet the stated criteria" is a one-shot classification, not a reasoning marathon.

Cheap gates protect expensive models. That one principle shows up everywhere in Composer, and it's most of why the cost numbers stay sane.

Honesty compels me to show you the escape hatch too. When code review hits max rounds without converging, Composer force-merges with a comment documenting the unresolved concerns. That's not a feature I'm proud of — it's empirical evidence that AI reviewing AI doesn't reliably converge. Two models will happily volley polite disagreement forever. The deterministic core decides when the conversation is over, because somebody has to, and it turns out the somebody works best as a counter, not a vibe.

Letting Go of the Provider Zoo

Now the confession.

On June 15th, I shipped a commit titled "feat(codex): finish integration — current models, full autonomy, agent-format fix." Weeks of work, done. Codex was fully integrated into Composer as a first-class agent provider, alongside a small zoo of others I'd been collecting.

On June 29th — fourteen days later — I deleted it. All of it. Codex, Antigravity, Pi, the runners, the parsers, the quota providers, the UI. An eight-pull-request purge that collapsed Composer's entire provider system down to exactly one entry: Claude.

I wrote a whole post recently about holding your work loosely — about how the hard part of this era isn't learning new tools, it's releasing the things you've built when they stop earning their place. I published that on June 30th. I made the decision to delete the provider zoo the day before. Apparently I was writing myself a permission slip.

Here's what forced the decision. Every provider was its own runner, its own output parser, its own quota logic, its own set of edge cases — an integration surface that grew faster than the value it returned. That was survivable. What wasn't: the output quality gap. When I compared the work coming off the line, Claude Code — the model and its harness together — was consistently producing better results with less babysitting. An autonomous factory where some of the workers need babysitting isn't autonomous. It's a daycare with extra steps.

Multi-provider support was optionality I was hoarding, not capability I was using. Deleting it deleted a whole category of maintenance from my life, and the platform got better — every improvement now lands on one deeply-integrated engine instead of being watered down across four shallow ones.

Focus, it turns out, is also an architecture decision.

Cheap Models Where They Belong

Going Claude-only didn't mean going expensive-only, and this is the nuance I want to be careful with: this is not a story about which LLMs I use. The models are interchangeable labor. The routing policy is the interesting part.

Composer runs Claude through two channels: direct Anthropic, and open-weight models on Ollama Cloud — Kimi, Qwen, GLM, and friends — each channel with its own independently-tracked quota. On top sits an automatic model routing policy, the pattern the industry calls model cascading: classify the work, then spend accordingly.

Small tasks — docs, config, renames — route to GLM on Ollama Cloud first, with Sonnet as fallback. Medium tasks prefer Sonnet, with GLM as quota relief when the Anthropic window is tight. Large or high-risk tasks — anything touching auth, schema migrations, concurrency, architecture — go to a frontier model, always, and are never silently downgraded, no matter how much quota pressure the system is under.

That last rule is the whole policy in miniature. Composer's model-capability doc states it plainly: when genuinely uncertain, prefer the stronger tier — a wasted frontier run is cheaper than a botched implementation. Cost optimization that creates rework isn't optimization. It's a loan.

The internal machinery follows the same logic. The acceptance-criteria judge is Haiku, because verdicts are cheap. The retrospective validator — coming up next — is pinned to Opus, because it's a trust decision. Every AI invocation in the platform has a deliberate answer to how much model does this decision deserve?

The dashboard keeps the receipts: 707 tasks through Anthropic direct for $3,450, and 86 tasks through Ollama Cloud for $589. The cheap channel is real, it's earning its keep — and the expensive channel is doing the work that justifies its price.

The Factory Files Bugs Against Itself

This is the newest addition, and it's my favorite thing Composer does.

After every merged pull request, a read-only retrospective agent reviews the completed task — the original prompt, how many revision rounds it took, what the reviewer flagged, the final diff — and files improvement issues for what it finds. Improvements to the target project go to that project's repo. Improvements to the workflow itself get filed against Composer's own repository.

Read that again: the platform reviews its own performance and opens tickets against itself. So far it has filed 39 of them.

Filing issues is observation. The loop closes with Retro Auto-Queue: an admission poller scans those retrospective issues and — within strict limits — turns them into queued tasks automatically. Each candidate first goes before a validator agent, pinned to Opus, which checks the proposal against the live codebase and returns approve, reject, or defer. Rejections happen regularly. The validator having teeth is precisely what makes leaving it switched on defensible — I've watched it kill ideas that sounded plausible and would have been wasted work.

A self-improving system with no limits is a runaway process wearing a nice name, so the guardrails are the design. Every retro issue carries a generation label — work spawned by a retrospective on organic work is generation one; a retrospective on that work files generation two. Anything past generation one never auto-queues; it waits for me. There's a daily admission budget, a concurrency cap of one, and a detail I genuinely admire: the budget meters validations, not approvals — so a string of rejections can't cause unbounded Opus spend. The safety rails are deterministic. The inverse architecture, applied to the scariest feature in the building.

Does it work? Here's a real chain from last week. A generation-one retrospective caught a genuine bug: the auto-review artifact was duplicating the entire review. The fix merged. The retrospective on the fix then filed a generation-two issue with a sharper observation: the bug had to be fixed in three places, because the message-pump loop was copy-pasted in triplicate — consolidate it. That refactor merged too. The retrospective on that filed a generation-three issue: document the testing gotcha that had confused reviewers along the way, so it never bites again.

Found the bug. Fixed it. Critiqued its own fix. Refactored the root cause. Wrote the documentation. Three generations of compounding improvement, and my total contribution was approving the parts that crossed the generation line.

The Night Shift Nobody Asked For

The other new subsystem is a scheduler — recurring jobs that bind a project, an agent, and a skill to a cron expression. Pre-flight validation confirms the referenced skill still exists before every fire; broken jobs get flagged instead of failing silently.

I'll be honest about its maturity: the plumbing shipped recently and the dashboard shows exactly one scheduled run so far. But the reason I'm highlighting it is where it points. Composer supports custom agents with custom skills, which means scheduled work doesn't have to be software work. Newsletter generation. Social media engagement. Weekly retrospective sweeps across every project. A factory that can hire any specialist and put them on any shift isn't a coding tool anymore — it's programmable capacity.

The possibilities genuinely feel endless, which is exactly why I'm forcing myself to grow this one slowly.

What 793 Tasks Actually Cost

I promised numbers, so here's the dashboard, unvarnished:

A few of these deserve a second look. The 200:1 ratio between tokens in and tokens out is context engineering made visible — the overwhelming majority of spend is the system reading and re-reading code so that agents write the right thing, not generating text. The 23-minute average is the number that quietly changed my behavior: when idea-to-merged-PR is a lunch break, you stop rationing your ideas.

The one that matters most to me is the 94%. Reliability is the entire game. A factory that ships confident garbage fast is just a faster garbage factory — accurate and relevant is the bar, and the loops in the middle of this post are what hold it there.

For scale: those numbers span real production work — Leaderboard Fantasy features, TutorPro development, and 268 tasks Composer ran against its own codebase.

The Gap I Haven't Closed

Time for the honest section.

Autonomous test automation is still the hole in the factory floor. My agents write and run unit and integration tests, CI gates every merge — but nobody in the pipeline drives the actual application the way a QA engineer would. Browser-based end-to-end testing, role-based scenarios, the weird edge cases that live between components: still absent. I flagged this gap in the original post four months ago, and I won't pretend otherwise — it's still open. I have no committed approach yet. Computer use — agents driving real browsers and desktops — is advancing fast and may end up being the accelerant, but "may" is doing honest work in that sentence.

Digging into why I haven't closed it taught me something about where humans actually remain load-bearing in a system like this. I'm still the UAT gate not because the models can't click through my apps — they increasingly can — but because final acceptance keeps collapsing into questions of judgment. Does this interaction feel right? Is this detail consistent with where the product is heading? The vision lives in my head, and I haven't yet found the discipline to externalize it into criteria an LLM can reliably hold.

In the current taxonomy, I've moved from human-in-the-loop — approving every step — to human-on-the-loop: the system acts autonomously and I supervise outcomes at the edges. My remaining role isn't oversight of the code. It's custody of the taste. For now, that's the honest boundary of the factory — and closing it is less about better models than about me doing the spec-writing on my own judgment.

What I'm Chewing On

Two threads are pulling at me for what's next.

First: getting execution off my desk. Composer's brain is a workflow engine, but its muscle is my Mac — agents run as local processes, capped at a few concurrent slots. The orchestration layer wouldn't care if the muscle lived somewhere elastic. The options I'm circling: Google's Gemini Enterprise Agent Platform and its Agent Engine runtime, which keeps everything in the GCP stack I already deploy to; Claude Managed Agents, which is essentially the Claude Code harness as a hosted API — conceptually a one-to-one match for "dispatch a coding task, collect a result," though it's still in beta; or the smallest possible change — keep my orchestrator exactly as it is and rent only the sandboxes, spinning up isolated environments per task on something like E2B, Daytona, or Fly.io. The deterministic core makes this an implementation detail rather than a rewrite: when the core owns all state, the executor is swappable. The inverse architecture pays for itself again.

Second: making the tokens smarter. Remember that 5.16 billion tokens in? A meaningful slice is agents re-discovering the same codebase structure, task after task. The pattern I want to steal from the current wave of tooling is the code knowledge graph — index the codebase's structure and relationships once, let agents query the map first and only then navigate the territory. The emerging consensus says it well: coding agents need codebase maps, not bigger prompts. A local graph fronting every large repo could cut a real fraction of that input spend — and make the agents faster and sharper at the same time.

If either experiment produces results worth sharing, you'll read about it here.

We're Way Past Vibe Coding

In the original Composer post, I described the early days honestly: "I had a loose idea and I started vibing." That was accurate then. It would be a lie now.

The timing is almost too clean. Andrej Karpathy coined "vibe coding" in February 2025 — fully giving in to the vibes, forgetting the code exists. Almost exactly one year later, he declared that era over himself and proposed a better name for what comes after: agentic engineering. His framing is the one I'd stitch on the factory wall: vibe coding raises the floor; agentic engineering raises the ceiling.

Raising the ceiling doesn't look like better prompting. It looks like systems: deterministic cores and gated AI. Review loops with budgets. Validators with teeth. Routing policies, generation caps, quality bars you can graph. Engineering discipline wrapped around stochastic workers — the same profession we've always practiced, aimed one level up.

That's the invitation I want to leave you with, because this post was never about my platform. Composer is one developer's answer, shaped by my constraints and my stubbornness, and I'd genuinely love to see yours. So here's the challenge: pick the part of your process you repeat the most, and instead of doing it again — build the thing that does it. Give it a deterministic spine. Let AI in only at the joints, only where judgment lives. Add a loop that checks the work and a budget that bounds the loop. Then stand back and see how it changes what you're willing to attempt.

Stop thinking about your next feature. Think about the system that ships your next hundred.

The software is becoming the easy part. Build the thing that builds it.

–Jeremy