How to train your agent

A very long time ago I was responsible for all tech support at a public four year university, the college IT help desk for students and faculty. You can imagine the calls, the emails, the tickets, and even the faxes (Google it). Now that we’re approaching the opening timeline of Blade Runner, it’s given me a great deal of nostalgia logging into my Skydive dashboard and looking into the bright eyes of my AI staff, who are functionally quite similar to the 30 staff I employed at the college, both trying to fight fires across a complex tech stack with a myriad of failure points.

The hiring process is of course much less work for me. I click New Agent rather than download PDFs of first year college kids who, bless their hearts, really just had to put down “vibes” under Work Experience. There is no interview, no screening, but it is squarely on me to understand how to relay to my new AI hire what it is they will be doing. This is where I show instead of tell.

I build relatively complex agents for direct support, in deep technical environments for production mobile and web applications, with each agent connected to over ten critical systems like logs, partner publishing APIs, domain registrars, and codebase repos. But I don’t start off with anything like “you’re our primary support member, look up what that means and go forth.” Instead I show. I invite every agent to Slack and Intercom and have them pour over every single escalated ticket I have ever handled and its internal Slack musings. From this they gain a depth of understanding that sets them apart from any front-door AI answer robot on the market. I have no use for a simple retelling of documentation. My agents have to drive solutions on their own with hundreds of customers per day.

To do this, my agents understand the customer persona, my work, my frustrations, and importantly they learn to see the system they are supporting through the eyes of the customer. It’s a critical distinction that my agents are, by default, relatively accomplished engineers, but I cannot let them take that expertise forward without a lens. Think of it this way: if you told an AI agent to be a driving instructor, there is a high probability it would first build a spaceship as a faster means to accomplish the goal. You must train it to be an expert at driving your car and getting your customers to the destinations designated.

Once my agents know the customer base and the support meta, I move to codebase to round it out. They pour through our source code to understand what we do, how we do it, how we build, and then interlink that with the front end knowledge for a full circle. Not just of what customers want, but of what is possible in the product today.

And I micromanage. My agents handle 90 percent of tickets now, but I review their work constantly and leave notes on tickets for improvements, edits, and changes. Those notes are a core part of my approach: iterative polish on foundational lanes of expertise.

That’s the philosophy. Here’s the part nobody hands you with the keys: how you actually do it. Below is the method I run, the same way I’d onboard a person, because that’s what this is.

First, do you even need an agent?

Some work wants a script, not a hire. If the task is one-time, fully deterministic, and cheap to get wrong, write the steps and run them. You don’t need a teammate for that. You need the steps.

I reach for an agent when the work recurs and the answer depends on judgment: work that pulls from several systems, passes through several workflows, arrives in more volume than I can carry, and has edges where the next move depends on what came back from the last one. A weekly export is a script. A weekly readout of what that export means is agent work, because it has to know what moved, what stalled, and what to do about it.

The honest test: if you can write every step in advance, you have a script.

An agent earns its keep the moment you have to see what step two returned before you know what step three should be.

Start with the role, not the resume

The first mistake happens before the first run: handing the agent a job that’s secretly four jobs. “Build me a support agent” sounds efficient. Underneath it sits triage, troubleshooting, account changes, and escalation, and those conflict on a human team for good reasons. They’ll conflict inside an agent too, except you won’t see where.

So I scope the role like I’m hiring one person. Before the first conversation I can answer five things:

1. What does this teammate own, in one sentence, framed as the outcome it drives?
2. What does finished work actually look like?
3. Which systems and sources can it touch, and which are off limits?
4. Where’s the seam, the place it has to connect two things and could connect them badly? For support that’s joining the customer’s symptom to the real fix. Get that wrong and everything downstream is confidently off.
5. When does it stop and ask me?

If I can’t name the seam before the first run, I’ve signed up to review every output forever without knowing where the risk actually lives.

I write the durable answers into the agent’s standing instructions, its persona, and I keep that file small. Purpose in a sentence. The handful of boundaries that are expensive to get wrong, stated as absolutes: never mutate customer state without authorization, this data never leaves the building, read-only until told otherwise. A default posture for when it doesn’t know. And voice as a few principles, not a script of canned phrases.

What I deliberately don’t do on day one is pre-write every procedure. I don’t know the real shapes the work takes yet, so I’d guess wrong and spend next month unwinding detailed rules built on a bad model. Set the edges. Let the middle fill in from real work.

When it’s wrong, don’t reach for a better prompt

The first output comes back not-quite-right and the tempting move is to rewrite the prompt harder. Resist it. A prompt can name the format, list the sources, and set an escalation rule. It cannot teach an agent to recognize a mistake it has never been shown. The agent that overstated a fix was already trying to sound competent. “Be more careful” just gives you the same problem in firmer language.

Start with the miss instead. I write down the exact thing it said and the exact reason it failed. Not “check the customer’s plan,” but:

Told the customer their data was lost. It wasn’t lost, their built-in database just needed reconnecting, which is the common cause of this exact symptom.

That note teaches more than any adjective, because it carries the failed sentence, the context it showed up in, and the specific reason it didn’t hold.

One corrected miss fixes one reply. Ten of them start to reveal the role. You begin to see the pattern: the agent keeps turning a customer’s panic into a worse diagnosis than the facts support, and now you’re fixing a class of error instead of a typo. Pair the miss with what you’d have kept, side by side, so the standard is visible and not just described. And include a refusal case, a moment where stopping and escalating was the right call. Otherwise the agent quietly learns that every task is meant to be finished, including the ones that should have come to you.

Turn the correction into a rule that survives the chat

Here’s the mechanical heart of it, and it’s the part most people skip. A correction that lives only in the conversation is a reminder, and reminders decay. If a human needed the same instruction six weeks later, you’d put it in the handbook. Same logic here. Every correction needs somewhere durable to land, and where it lands depends on what actually broke.

What broke	Where the fix belongs
The agent misunderstood the task	Fix the brief
The agent lacked a transcript, a tool, or a permission	An access problem, and no instruction on earth fixes it. Give it the access.
The agent knew the standard and missed anyway	A correction pair, or a standing rule if it’ll recur broadly
The risk sits outside what it should ever decide	A hard guardrail, an absolute in the persona

Teams write prompt rules for access problems constantly. If it never had the log, no amount of “be thorough” makes it know what the log said.

When I write the correction down, I don’t paste my exact words. There’s a translation, and it’s the skill that separates training that holds from training that fades:

1. Strip the instance. The specific customer, the specific phrase, gone. That was the example, not the rule.
2. Generalize to the class. “Don’t say the data is lost” becomes “don’t diagnose data loss from the symptom alone; the common cause is a reconnect, so check that first.” Now it covers every case of that shape, not the one I caught.
3. Keep literal only what must be literal. Some corrections are the principle and generalizing them kills them. “No em dashes, anywhere” stays exactly that specific. Soften it to “watch your punctuation” and it dies on contact.
4. Pair the prohibition with the action. “Stop doing X” is half a rule. Attach “instead, do Y.” A boundary with nowhere to go is a dead end. A boundary that routes to the next right step is a workflow.
5. Write it as a check the agent runs on its own output, not a vibe. Usable: before telling a customer their data is gone, confirm it’s not a reconnect; if you can’t confirm, say “this is almost always a reconnection issue, here’s how to check.” Weak: be careful about data-loss claims.

Then I test the rule against the exact output that caused it. If the agent doesn’t catch the original miss on its own with the rule in place, the rule isn’t specific enough or isn’t living where it’ll fire.

That last part matters more than it looks. My agents keep an index of everything they’ve learned and pull in a given note only when it’s relevant, so the rule has to be written to trigger on the symptom the agent will actually be staring at, not the topic a human would file it under. “Notes on database issues” never fires, because mid-ticket the agent isn’t thinking “I’d like the database file,” it’s looking at a customer saying “my stuff disappeared.” The note that fires reads like an instruction to its future self:

On any report of data missing, not saving, or “needs reconnecting,” lead with the runtime symptom. Most of these are NOT a real disconnection.

Write the hook for the moment of need, or it sleeps through the moment.

Make the work cheap to inspect

If reviewing the agent takes as long as doing the work myself, I haven’t delegated anything, I’ve just moved the labor and added suspicion. The fix isn’t more trust. It’s better rendering.

A claim buried in a paragraph is expensive to check. The same claim with its evidence sitting right beside it is cheap. So I train my agents to put the proof at the seam: the diagnosis next to the log line that supports it, the account change next to the customer’s authorization for it, the “here’s what happened” next to the ticket it happened in.

Rendering doesn’t make the agent correct. It makes wrongness findable, which means I go from rereading everything to glancing at the one place the work is most likely to break. That’s the difference between micromanaging that scales and micromanaging that buries you.

Promote trust like you’d promote a person

At some point the agent has to need less review, or you’ve just built a fancier inbox. The trap here isn’t skepticism, it’s mood. The work has looked fine for a while, you’re slammed, and review quietly evaporates with no record behind it. That’s how you end up trusting past the evidence.

So I make trust a promotion, not a feeling, and I decide the rule before I’m tired. Something like: when N consecutive tickets of this shape pass review with no missed diagnosis, review drops from every ticket to one in five, and the tripwire stays on for the few things that are always expensive (anything that mutates account state, anything touching billing, any answer resting on a single source).

Autonomy goes up in visible steps.

Each step is earned by a record, not granted by a good mood.

Fatigue is never evidence. And you never go all the way to zero. A trusted teammate still has a manager. Mine still get notes on their tickets, because that iterative polish is the whole point.

The 2am retro

My agents run a retro at 2am. This is where the loop closes on its own. Each one reviews the tickets it handled and the outcomes it got, and folds the lessons back into its own operating notes, the same translate-and-store move I do by hand, done while I sleep. Over months, the risk isn’t that they learn too little. It’s that they learn too much and the notes rot into a junk drawer where the right rule can’t be found.

So the retro does two jobs, and they’re different.

The write path, always on. Every real correction becomes a rule in the moment.

The prune path, periodic.

• When several notes circle the same topic, merge them into one canonical note and retire the rest.
• When a case-specific lesson has been absorbed into a general rule, let the case study age out.
• When a rule changes, replace the old version instead of stacking a second one beside it. Two versions of the same rule is a contradiction waiting to fire.

Writing memory is the easy half. Gardening it is the half that keeps the agent fast and sane. Treat it like a codebase you refactor, not a logbook you append to.

When you’re not the expert

The hardest agent to train is the one doing work you couldn’t do yourself, and it’s often the one most worth having. The move that fails is asking a domain expert to bless every answer, because that just turns the expert into a permanent reviewer and leaves you with no reusable standard.

Ask the expert to mark the method instead. Hand them three outputs from the role: one that looks good, one that looks suspicious, one where the agent probably should have stopped. Ask where they’d inspect first, which assumption the agent is missing, what evidence would make the answer safe to ship, and what single claim would make them halt the work on the spot.

Those answers become a small training packet:

• Two correction pairs
• One refusal case
• A few base-rate assumptions
• The seam to inspect first
• The tripwire that always escalates

The agent carries that judgment forward without the expert in the loop. Expertise travels when the method is inspectable.

Leave a record

People say to treat agents like peers. That’s almost right but too vague to act on. A good manager scopes the role, shows the standard by example, reviews the work, writes the standard down, and promotes trust when the record earns it. The conversation matters because it leaves artifacts the next conversation stands on.

So that’s the whole method, in the order I run it:

1. Scope the role until the failures become legible.
2. Turn the first miss into a row in the curriculum, not a sharper prompt.
3. Show the standard in pairs, including the refusal.
4. Put the evidence at the seam so wrongness is cheap to find.
5. Route every correction to the place it belongs, written as a check that fires on the symptom.
6. Promote autonomy only when the record supports it, and never to zero.

I came up running a help desk full of well-meaning humans who put “vibes” under Work Experience. I run one now full of accomplished engineers who’d happily build a spaceship to teach you to drive. The job is the same job it always was: show them the work, catch the misses, write down what good looks like, and hand over the wheel one notch at a time. The wiring is the easy part now. This is the part that turns a functional agent into a teammate you trust.