The pipeline was big, but about as simple as they come: data in, transform, data out, at around 450,000 messages a second sustained, bursting past three million. The only real requirement was that every incoming message get its source stamped into it on the way through. A trivial mutation. Three bytes appended to the end of the protobuf, which the wire format is perfectly happy to let you do, since a decoder reading tag-value pairs just takes the last value it sees for a given field. So I did the thing that looked easy. I wrote a WASM transform, loaded it into Redpanda, let it blindly append the three bytes, and moved on.

The transform was correct. It worked great! That was never the problem. The problem was that Redpanda Transforms are hot on paper, and in my case were the thing quietly destabilizing the whole cluster at the scale I was running at, and the logs - they told me nothing at the time. Three weeks of chasing instability that had no signal attached to it. When I finally pulled the transform and wrote the boring Go consumer I should have written on day one, the instability magically disappeared.

Here’s the part that stings (my ego). My gut said “just write the consumer” at the very start. I ignored it because the transform was less code and lived inside the broker, so there was no separate service to run and deploy. Both of those were true. Also, it’s WASM, transpiled from Go - fast as hell! Also true. But .. none of them were worth three weeks.

I’ve spent most of my career on the deciding end of calls like that one. Toolchains, workflows, libraries, occasionally the whole stack. It usually starts the same way - someone gets excited and drops a link in a channel, a new build tool or ORM or queue, whatever it is this week. Sound familiar? None of that is wrong. It probably is faster, the DX probably is better. But.. the part nobody leads with is what it costs, and who ends up paying. Which is why I keep landing back on my same soap box:

What meaningful value does adopting it provide, and at what cost?

That's it. That's my whole method. It sounds like the kind of thing you'd find in a slide deck, and the reason it isn't useless is that both halves are much harder to answer honestly than they look. I answered both wrong on a pipeline I was building alone, with my own gut handing me the right answer for free.

Value is what you can point at later

Value is the easy half to talk about and the hard half to prove (see: quantify), because most of what gets called “value” in the moment is essentially vibes. It’s less code. It’s cleaner. It lives in the broker instead of in yet another service I have to babysit.

Some of that’s real, and the WASM transform had genuine value (on the surface) - fewer moving parts, nothing extra to deploy, the mutation sitting right where the data already was. I’m not going to pretend those were imaginary - they’re exactly the kind of alluring feature that wins an argument in your own head at ten in the morning.

The test for whether you’ve found real value is whether you could show it to someone in six months. Not describe it. Show it. And the value that actually held up in that whole mess wasn’t on the transform’s side at all. It was on the boring consumer’s - a stack trace when something broke, standard tooling, logs that said words. That’s developer experience as a line item instead of a vibe. The transform felt better to write. The consumer was better to live with, and I could prove it, because one of them left me blind and with a headache for three weeks, while the other didn’t (even though it took longer to spin up).

Cost is almost never money

Ask a developer what something costs and they won’t say dollars. They’ll say time, complexity, risk, and how much worse their week gets. All of it converts to money eventually, yes, but slowly enough that nobody has the conversion in the moment.

Time is the one that gets underbid, and it’s not implementation time. Implementation you can estimate. What you can’t estimate is time-to-understand - the stretch where you’re slower because you’re learning a new way to solve a problem you already knew how to solve. The usual story is that the person proposing the change has already paid that cost, so their estimate is honest and also wrong, priced from the far side of the curve.

What my three weeks taught me is that it’s worse than that. I was the person proposing the change and the person paying for it. There was no over-eager colleague to blame. I’d done the reading, I understood transforms, and I still priced the decision from the wrong side, because “I know how this works” and “I know what this costs to run” are not the same knowledge and I treated them like they were.

Then there’s risk, which is where the unknown unknowns live, and which you cannot estimate by definition. Cluster instability with no diagnostic signal is what an unknown unknown looks like when it finally shows up - not a bug you can point at, just a system that’s unhappy and won’t tell you why. You don’t get to estimate that in advance. You only get to decide how much of it you’re willing to eat, and I ate a lot of it without meaning to.

And assessing all of this has its own cost. Sometimes the evaluation is more expensive than the thing is worth, and the right move is to not evaluate it and go with the boring option you already know and trust. That feels like laziness, which is why people don’t say it out loud. It would have been the correct call here.

The guideline is somebody else’s context

Community guidelines are guidelines. Best practices are best-for-somebody, and that somebody wrote them up because it worked for them, which is the best possible reason and also not a reason for you.

WASM data transforms are the modern, promoted way to mutate in-stream. That’s not marketing, it’s genuinely a nice pattern, and for plenty of teams it’s the right one. The docs point you at it. The blog posts point you at it. Everything points you at it, and none of that was evidence about my situation.

My situation was one engineer, one trivial mutation, 450k messages a second sustained, and a cluster whose stability was the entire ballgame. Drop compute into the broker at that throughput and the broker’s stability is now downstream of my compute, which is a trade a bigger team with more eyes and better broker-level observability might make on purpose. I made it by accident, because the pattern was the recommended one and I didn’t stop to ask which context produced the recommendation. If you can’t articulate the situation a guideline came out of, you can’t tell whether you’re standing in it.

The scale isn’t balanced, and it shouldn’t be

I’ve been describing this as a scale, value on one side, cost on the other, and I want to break that on the way out, because a scale implies the two sides start even. They don’t.

The thing you already have is paid for. In my case the thing I already had was ~5 years of knowing exactly how a Go consumer behaves when it’s unhappy at scale. The learning curve was behind me, the failure modes were boring and documented in my own head, and I threw all of that away for less code and no extra deploy. The new thing gets charged full price, and it gets charged later, and here it got charged to the one person who’d explicitly warned himself not to do it.

So the burden of proof sits on the change. Not because new things are bad, and not because I’m precious about the stack. But because the cost of the new thing is systematically underestimated by exactly the person most excited about it, and I’ve now been that person and that person’s only victim in the same three weeks. The value is loud and early. The cost is quiet and late.

Ask what it’s worth. Then ask who’s paying, and when. And if your gut already answered, at least make it argue with you before you overrule it.

And that’s the whole ~thing in one moment. I spent a good while building a dev harness for Claude Code, that I call core-dev, and the part I keep wanting to talk about isn’t the feature list. It’s that the thing has opinions, that I codified and force it exert, and it’s built so those opinions are expensive to ignore. It loads the skills a turn needs before I ask (a lot more reliably than out of the box Claude does). It blocks a commit of code nobody reviewed, or a secret I didn’t notice I’d staged. The judgment calls it leaves to me; the mechanical ones it takes off the table. Most of the time it stays out of the way, and when it doesn’t, it’s usually because I was about to do something dumb.

I’m writing this series because every one of those behaviors is a design argument, and I’d rather defend the arguments than document the buttons. It’s also the one thing most agent-tooling posts don’t offer right now: these aren’t predictions about how you should build a harness, they’re claims with a running system behind them. I built the thing, I’ve been wrong inside it plenty, and it pushes back on me in ways I want to show you.

You don’t need my plugin for any of this to make sense, I hope.. The questions underneath are the ones you hit anywhere an agent does the work - where a rule should actually live, what you let a model decide, what you never trust it with, and how you stop the agent that writes code from also being the one that runs it. I came down somewhere specific on each, and the posts are me arguing for where I landed.

Here’s the map, in order. I’ll update links to new posts as I get them published.

Cheers!

Make the right thing the default, and the wrong thing loud. You get better behavior by reducing the cost of the “right path”, not by forbidding the wrong one (hey, sometimes I just feel like stabbing myself in the eye, okay?)

Adherence is a budget. Every rule you write taxes every other rule, so route each one to the cheapest mechanism that will actually enforce it.

Plan, build, review. One feature walked from the first prompt to an open MR, with every guardrail firing in order. This is the guided tour, if you want the whole shape in one read.

A number is not a fact. A measurement isn’t mechanical just because it has a decimal point. When to gate, and when to \shrug.

The opinion with a decimal point. I built a check that looked like signal, measured it against reality, watched it predict nothing, and shipped the negative result straight into production.

If it isn’t written to disk, it didn’t happen. Just like PCAPs (or it never happened). Every step starts from nothing, so the only memory that survives is what gets written to a file.

No context holds both a shell and a pen. The one rule that is the security spine: the context that can write can’t run, and the one that can run can’t write.

The harness that tests itself. It fails open at runtime and fails closed in CI, and pointing its own review at itself caught a real bug.

Let the model decide only what it can’t get wrong. Hand the model the one judgment call it’s actually needed for, and wrap deterministic code around the rest.

Lean into the gate. A denied action isn’t the tool being difficult. It’s naming the step you skipped.

New pieces go up about once a week. If that sounds like your kind of thing, subscribe and I’ll send each one as it lands.

Then it got a task where the spec was wrong.

The executor did exactly what it was told, the reviewer correctly said no, and the executor went back and did the same wrong thing again. Only, slightly differently.. Until the turn budget ran out. \sigh

Nobody was confused. Every node did its job. The problem was that the only edge out of the reviewer went to the executor, because when I drew the thing, why would there be an edge back to the planner? The planner had already planned. That was step one. We were past that.

And that’s the bug, and it isn’t in any of the nodes.

A script terminates when it runs out of steps. An agent replans when it runs out of steps.

That’s the whole difference. Everything else is plumbing we’ve all built a dozen times: a queue, a worker pool, a state store, retries with backoff. If you’ve written a job runner, you have all the parts. What you’re missing is one loop boundary in a different place.

I want to be clear that the pipeline instinct isn’t wrong. It’s the correct first move. It’s legible, it’s cheap, and for most of what people call “agents”, it’s also the last move they should make. The pipeline stops being right at a specific moment, and it’s worth being able to name that moment rather than reaching for the loop (because loops are what serious people build).

The call graph is the problem

Here’s the reframe that took me an embarrassingly long time to wrap my head around: the agents never call each other.

That’s literally it - if there’s one thing you take away from this post, it’s that. Internalize it.

The Misnomer

“Agent-to-agent” is a misnomer that smuggles an architecture in through the name. It sounds like a protocol between peers, right?

Well, it’s not. It’s agent, then state, then agent. Nobody’s phone rings. We’re not passing notes in math class - each node reads from a shared state, writes an artifact back, and a dispatcher decides who runs next.

And thus, here we are . It’s not a DAG - it’s a work queue with a scheduler. Not so scary now, right? The only interesting difference is that one worker type is allowed to enqueue new work, and its enqueue decision is a model output instead of a branch you wrote at 2am on a Tuesday six weeks ago after a few beers (not unlike this post).

That’s what the “edges” are, by the way. Every edge in your DAG is a decision frozen at authoring time. if review_failed: retry() is a guess about a situation you had not yet seen, made by a person who had less information than the system will have at the moment it matters. My reviewer-to-executor edge was a guess that review failures mean the code is wrong. Usually true, yes. But .. not always true. And the DAG has no way to find out, because the guess is load-bearing structure.

Delete the edges and the decision moves to runtime, where the information actually is.

Four node types, and one rule for placing them

Four kinds of node: planner, executor, verifier, reviewer. Their signatures tell you most of what you need, and the rule for placing them is simple.

If a node’s output is a decision, it’s frontier.

If it’s an artifact with a deterministic check, it’s the cheap model.

Decisions aren’t testable. That’s not a limitation to engineer around, it’s the criterion. “Should we retry or decompose?” doesn’t have an assert. But “Does pytest tests/test_auth.py::test_expired_token exit zero?” does. Route on that property, and the split falls out of the architecture instead of being a cost optimization you bolted on afterward.

Planner: state -> [task]

The only node with agency. It reads the entire history and decides what happens next, which makes it the one place in the system where “it depends” is an acceptable answer.

Two things people get wrong here. It runs on empty queue, not on schedule, and not on failure. There’s no on_error: replan() hook, because replanning isn’t an error path, it’s the normal path that happens to also cover errors. And it doesn’t plan the whole job. It plans the next chunk, which is usually one task, occasionally three, and never seventeen. A planner that emits seventeen tasks has quietly turned back into a DAG author, except now the DAG is written by something with worse judgment than you and no ability to draw it on a whiteboard.

Use the best model you can afford. This is where the money goes.

Executor: task -> artifact

Takes one task, produces one artifact. Makes no decisions, routes nothing, and never gets asked what should happen next.

Cheapest model that can hold a tool loop without falling over, which is a lower bar than it was a year ago and is the entire subject of the follow-up post. I’m a huge fan recently of protoLabs' quant of Ornith 1.0 35B @FP8

Verifier: artifact -> verdict

The one I’d tattoo on people: the verifier is code. Sear this into your memory. It is not a model. Not a model with a really good prompt. Not a model with a really good prompt and structured output. It. Is not. A model!

The instant the verifier is a model, you’ve rebuilt self-assessment, and self-assessment is the thing the whole apparatus exists to avoid. Done means verified deterministically or it doesn’t mean anything. An executor that grades its own homework is a very expensive way to generate confident nonsense, and it will do it politely, in JSON, with a "confidence": 0.946.

What this looks like in practice is unglamorous, and that’s the point. A shell command and an expected exit code. A schema validation. A diff that has to touch a file in the allowed set. grep -c on a log. If you can’t write the check, you don’t have an acceptance criterion, you have a vibe, and the planner shouldn’t have emitted that task in the first place.

Reviewer: artifact -> judgment

Reads artifacts that already passed the gate and forms an opinion about them. Use a frontier model, because opinions are decisions wearing a different smoking jacket.

A judgment is not a verdict and it’s worth keeping the words apart. A verdict is mechanical: the test passed, the build is green, the schema validated. It’s a fact, and code produces it. A judgment is an opinion, and opinions come in flavors: the fix mutes the symptom instead of curing it, the abstraction won't survive the next feature, the code is technically correct and morally a crime. Both are useful. Only one of them is allowed to gate anything.

The reviewer doesn’t route either. It records, same as everyone else, and the planner reads the record on the next tick and decides whether “morally a crime” is worth another three tasks or whether we ship it and open an issue.

The loop

Take a look at this conceptual python.

def run(goal, budget):
    state = State(goal=goal)
    queue = deque()

    while not goal_satisfied(state):
        if budget.exhausted():
            raise BudgetExhausted(state.summary())

        if not queue:
            tasks = plan_next_tasks(state)
            if not tasks:
                raise NoViablePlan(state.summary())
            queue.extend(tasks)

        task = queue.popleft()
        worker = worker_for(task.capability)
        artifact = worker.run(task, state.context_for(task))
        verdict = verify(artifact, task.acceptance)

        state.record(task, artifact, verdict)
        budget.charge(worker.cost)

    return state

Now look at what isn’t there.

There are no edges. There’s no if verdict.failed: retry(). There’s no routing table, no conditional handoff, no state machine enum with AWAITING_REVIEW in it. When verify fails, that just gets recorded. The queue drains. The loop comes back around, finds the queue empty, and asks the planner what to do about a state that now contains a failed verification. The planner might retry it. It might decompose it into three smaller tasks. It might notice the spec was wrong, which is the thing my DAG couldn’t do at any price.

The retry loop is emergent, not coded. That’s the self-driving property, and it costs exactly one if not queue:.

I find this genuinely hard to hold onto. Every time I read this loop I want to add an edge. Some case comes to mind, the reviewer rejects on style versus correctness and surely those should route differently, and my hand goes to the keyboard. Don’t. That’s a decision, decisions are the planner’s job, and the planner has strictly more information at tick 12 than you have right now.

State is the architecture

Two properties, and both of them are load-bearing.

Append-only, not a mutable blackboard. The planner’s next decision is a function of history. Overwriting is amnesia. If tick 4 recorded “tried the obvious fix, test still red” and tick 9 overwrites it, the planner at tick 10 will happily try the obvious fix. You also get replay for free, which matters the first time one of them does something whack at 3am and you need to know whether the planner was wrong or the state was.

context_for(task) is a projection, not a dump. Each worker sees what its job needs. The executor gets the spec, the file paths, the test command. It does not get the planner’s reasoning, the reviewer’s prior judgments, or the transcripts of its five siblings.

This is where most implementations die, and it dies in disguise. You dump full state into every call because it’s easy and it’s correct-ish, and by tick four you’re feeding the executor a novel. Quality degrades. It feels like a model problem. You go swap models. It’s not a model problem, it’s that you handed a worker fifty thousand tokens of other people’s thinking and asked it to find the one line that was actually its assignment.

Stopping

Three mechanisms. You need all of them, and they fail differently.

goal_satisfied is a predicate over recorded verifications. Never a model announcing success. Same reasoning as the verifier, higher stakes, because this one’s the exit.

Budget exhaustion raises, and it raises with a state summary, not just a log line. Something a human can read to find out what happened.

No-progress detection: if the state hash hasn’t moved in N ticks, kill it and escalate. This is the one people skip and it’s the one that saves you money, because a planner stuck in a loop of semantically-identical tasks will happily burn every token you’ll give it while producing state that’s byte-identical tick over tick.

The principle underneath all three: an agent that can’t finish has to say so loudly. Silent success is the worst failure mode you can ship, because it’s the one that ships. A crash gets fixed on Tuesday. A green checkmark on a job that quietly did nothing gets found in November.

Watching it work: incident to patch

Alert fires. Goal: figure out what borked and fix it.

Tick 1. Queue’s empty, so the planner runs. It reads the alert, the logs, and a metrics window, and emits one task: write a failing test that demonstrates the reported behavior. Not “fix the bug.” The planner doesn’t know what the bug is yet and it isn’t pretending to.

Tick 2. Executor takes it. Reads the code, writes a test, runs it. Artifact: a diff containing one new test. Verifier is a shell command: does the suite fail, and does it fail in the new test rather than somewhere else? Say it passes instead, meaning the test went green immediately. Recorded as a failed verification. Not an error. A fact.

Tick 3. Queue’s empty again, planner runs again, and now it’s reading a state that says “our first theory of the bug produced a test that passes.” That’s information. It emits a different task: check whether the alert is even real, or whether the metric threshold moved.

The DAG I would have drawn cannot do tick 3. It would have taken the passing test as a green light and gone on to patch a bug that wasn’t there.

Note the gate sitting between the repro executor and the patch executor. Two executors back to back with nothing in between compounds error: the second one builds on the first one’s work with no independent check that the first one’s work was real. The verifier isn’t ceremony, it’s the thing that keeps the second executor from inheriting a hallucination as a premise.

Run the same alert twice and you’ll get different task sequences, because the planner sees different state. That’s the payoff and it’s also the part that makes these systems annoying to test, which is a real cost and I’m not going to pretend otherwise.

When you shouldn’t build this

Most of the time. Despite the echo chamber of modern .. life.

If the task shape is known, the steps are fixed, and failure means stop, just write the damn script. Tell your director to shove it and punch them in the face. It’ll be faster, it’ll cost less, you can reason about it, and when it breaks you’ll know why in about ninety seconds. Nobody has ever been paged at 3am because their bash script had insufficient agency, or it hallucinated a glob as rm -rf / in a privileged environment.

The loop earns its complexity in exactly one condition: the task space is open and partial failure is expected and informative. Open means you cannot enumerate the paths ahead of time. Informative means a failure at tick 2 should change what happens at tick 3. If failures don’t change the plan, you don’t need a planner, you need a retry decorator and a good night’s sleep.

Most things sold as agents are scripts wearing a hat. That’s frequently correct. The hat is the problem, not the script.

The tell

Here’s how I know now, before I write anything: I ask what happens when the plan is wrong.

If the answer is “it fails and I look at it,” write the script. Genuinely. That’s a fine answer and it’s the right answer more often than the discourse suggests.

If the answer is “it needs to notice and try something else,” you don’t want a better DAG. You want a loop with a planner in it, and you want the courage to leave the edges out, which is harder than it sounds because the edges feel like engineering and the empty space feels like you forgot something.

You didn’t. That’s the design.