From pilot to production: a 20-point checklist for AI agents

The demo dazzled the board, and Monday looked like Friday. That sentence describes most AI pilots in the mid-market, and the cause is rarely the model. It is that nobody settled, before the pilot started, what production would require. The twenty checks below are what KOI settles in the scoping week and the build. Use them on your own pilot, whoever built it.

Scope

1. 1.One named process. Not "customer service"; the triage and first reply on inbound tickets. A pilot with two processes has none.
2. 2.One metric and a recorded baseline. Hours per week, cycle time, error rate, or stockouts, from your own data, before the agent runs.
3. 3.One named business owner who wants the result and has the authority to change the process.
4. 4.What is out of scope, in writing. The exceptions the agent will hand to a person, and the adjacent work it will not touch.

Data and access

1. 1.An inventory of the systems the process runs on: CRM, ERP, mailbox, document store, ticketing, and the spreadsheets nobody admits to.
2. 2.Access granted through your existing permissions, scoped to the process. An agent with broader access than the person it assists is a finding waiting to happen.
3. 3.The data gaps named. Which records are incomplete, which are in the wrong place, and whether any of it blocks the result or merely limits it.
4. 4.Test data that is real. Last month's tickets, last quarter's orders, last audit's requests. An agent tested on invented cases is untested.

Governance

1. 1.Every action classified: read, prepare, act internally, act externally. The class sets the gate.
2. 2.An approval map: for each consequential action, the named person who approves it and the threshold above which it escalates.
3. 3.A policy check in the path of every consequential action: allow, route, or stop. Written in your terms, not the vendor's.
4. 4.A record of every action, its inputs, its policy outcome, and its approver, that an auditor could read without you in the room.

Testing

1. 1.Real runs with the real approvers in the loop, for long enough that they have corrected the agent's drafts and trust what they see.
2. 2.The failure cases rehearsed: a malformed input, a counterparty that does not reply, an action the policy stops, an approver who is away.
3. 3.Approvers trained, in the tool they already use, before launch rather than after.

Rollout

1. 1.Phased by volume or location: one store, one community, one practice group, then the rest on the same configuration.
2. 2.A runbook your team owns: what the agent does, what it hands over, who to call, and how to pause it.
3. 3.A fallback. If the agent stops, the process still runs the way it did before, the same day.

Measurement

1. 1.A monthly report against the metric from check two, in your numbers, from the first month in production.
2. 2.The next use case scoped before this one finishes, so the team that just learned to run an agent runs the second one sooner.

Why pilots stall

Run a stalled pilot against the list and the pattern is consistent. Checks one, two, nine, ten, and twelve are missing: no single process, no baseline, no action classification, no approval map, and no record. Those are not technical gaps. They are decisions, and they can be made in a week. KOI's scoping week makes them, and the free prototype that precedes it shows what the agent will do on your example before any of them are needed.