Practice
Agentic systems in production
I design systems that use models, and I run one in production that I own end to end. The interesting part is never the model. It is the set of things the system is structurally unable to do without a person, and deciding that before you build rather than after something goes out wrong.
- Running in production
- A full pipeline, mine end to end
- The design question
- What can it do unsupervised
- In an org
- Standards, and read-only tooling
Most failed AI projects I have watched did not fail because the output was bad. They failed because nobody decided in advance where the system was allowed to act, so either it could do everything and became too dangerous to leave running, or it could do nothing and became a demo.
The design question I care about is narrower than "what can the model do". It is: what is the worst thing this can do at three in the morning with nobody watching, and is that acceptable. If the answer is no, the fix is not a better prompt. The fix is to remove the capability.
The rules I build to
Five of them, each learned on something that shipped. They are in the code rather than in a policy document, because a rule that depends on somebody remembering it is not a rule.
A scheduled run can only ever produce a draft.
The failure mode of an automated publisher is not a bad sentence. It is a bad sentence that nobody saw. If the machine is structurally incapable of publishing, the worst case at three in the morning is a draft somebody deletes in the morning.
The dangerous action is a separate function, not a flag on the safe one.
On That Paleo Chick, publishing to the web and mailing the subscriber list are two different functions with two different call sites. If mailing were a boolean on the publish call, the whole list is one typo away from a recipe every morning. It is not a boolean, so it cannot be typed wrong.
Give the tools less access than seems reasonable.
The internal tooling I sponsored at Button was read only and tightly scoped, deliberately. An agent that can read an operational system and be wrong costs you a confused engineer. An agent that can write to one and be wrong costs you an incident. The second is not worth the convenience.
Validate structure, not quality.
A gate has to be able to fail. Checking that a post carries exactly one section tag, that its structured data parses, and that the topic is not already in the ledger are all things that either pass or do not. Asking a model whether its own output is any good is not a gate, it is a formality.
Put the approval where the person already is.
Approvals live in Slack because that is where the day happens. A dashboard that has to be visited becomes a dashboard nobody visits, and then the approval step quietly becomes a rubber stamp. Membership of the workspace is not the trust boundary either: the buttons that publish or spend money check an explicit allowlist.
Bringing it into an engineering org
01
AI-augmented delivery as the org default
I championed AI-augmented delivery until it was simply how the org worked rather than an initiative with a name, and helped shape the engineering standards for how it gets used. The standards mattered more than the enthusiasm. Without them you get a team where half the pull requests were written by a model and nobody has agreed what review means any more.
02
Read-only LLM tooling, scoped to prevent hallucination
I sponsored two internal tools at Button: an order debugger for a large retail partner, and a validator for the files partners send. Both were read only and both were given a deliberately small set of tightly scoped tools rather than general access, because a grounded answer to a narrow question is useful and a confident answer to a broad one is a liability. They took manual triage load off the team that was absorbing it.
03
Deriving service ownership from commits and reviews
Service ownership pages rot the moment a team reorganizes, and the cost shows up at the worst time, when an alert routes to nobody. I used AI over commits, reviews and pull request approvals to derive who actually owns each service from what people actually do, so alerting follows reality rather than a page somebody last edited two reorganizations ago.
04
Getting business logic out of people's heads
A handful of subject matter experts held logic that existed nowhere else, which is a staffing risk disguised as seniority. I ran a series of sessions with them and used AI to turn the recordings into documentation the whole team could work from. The AI did the transcription and structuring. The experts did the correcting, which is the part that made it trustworthy.
The worked example
The system I can show you end to end is the one I own outright: the content pipeline behind That Paleo Chick. Generation and image models behind validation gates, drafting into a CMS, and a human approval in Slack before anything reaches a public site or six social networks. Every rule above is enforced somewhere in it, in code rather than in a policy document.
It has its own page, because the interesting parts are specific and because it is mine to describe in as much detail as I like.
Stack
- Models and orchestration
- Claude, MCP, including custom servers, Multi-agent orchestration, Agentic PRD to ticket pipelines, Imagen 4, Gemini
- Control
- Human-in-the-loop approval gates, Read-only and narrowly scoped tools, Structural output validation, Explicit authorization allowlists, Append-only state
- Around it
- Node.js, TypeScript, Postgres, GitHub Actions, Slack apps, AWS