Human approval architecture for AI agents in regulated work
By Krish Subramanian, Chief AI Officer, KOI AI · Published · 4 min read
An AI agent in regulated work needs three things wired in before it touches a system of record: a policy check on every consequential action, a named person's sign-off on the actions that matter, and a record of every action that an auditor can read. Done well, approval is a review of drafts rather than a bottleneck. The agent prepares, the person decides, and the record shows both.
When a regulator, an auditor, or a board asks about an AI agent, the question is rarely "how good is the model?" It is "who approved this, and how do you know?" Companies that answer it with a policy document are in trouble. Companies that answer it with a record are not. The difference is architecture: the approval has to be built into the path an action takes, not written beside it.
Classify actions before you automate them
Every workflow an agent touches breaks down into four classes of action. The class, not the workflow, decides the gate.
| Class | Examples | Gate |
|---|---|---|
| Read | Retrieve a record, summarize a document, answer a question from the system of record | Access control only |
| Prepare | Draft a filing, assemble an evidence package, propose a purchase order, draft a reply | Policy check; a person decides whether it proceeds |
| Act internally | Create a ticket, update a status, file a note in a case | Policy check; approval by threshold and record type |
| Act externally | Send to a counterparty, file with a regulator, place an order, change money or a customer record | Policy check and a named person's sign-off, every time |
Four classes of agent action and the gate each needs.
Most of the value in a compliance, clinical, or financial workflow is in the first two classes, which is convenient: an agent can read and prepare at volume while every consequential action still waits for a person. In KOI's published engagements, the agents drafted compliance documentation, assembled audit evidence, proposed purchase orders, and captured clinical notes. People filed, sent, ordered, and signed.
Design the gate, not the agent
The gate is a small set of decisions made once and applied to every action:
- ◆Policy check: for each action, allow it, route it to an approver, or stop it and return it as a draft. The policy is written in your terms: amounts, counterparties, record types, hours of the day.
- ◆Thresholds: a purchase order under a limit goes to the buyer; above it, to a senior approver. A refund under a limit executes; above it, waits.
- ◆Routing: every approval goes to a named person, not a queue. Segregation of duties holds: the person who configured the agent does not approve its output.
- ◆Time limits and escalation: an approval that waits too long escalates rather than expiring silently.
- ◆What the approver sees: the draft, its sources, and the agent's reasoning, in the system where they already work. Approval that requires a new tool will not be done.
Record everything that matters
The record is the part most pilots skip and the part examiners ask for first. For every consequential action it holds what the agent proposed, what inputs it used, what the policy check decided, who approved it and when, and what then executed. A record with those fields lets you reconstruct any outcome, correct an agent rather than guess at it, and hand an auditor the trail instead of rebuilding it from inboxes. At a mid-size lender, KOI's accountability ledger served as the audit trail, and audit preparation went from weeks to days.
Keep it fast
Approval architecture fails when it is slow, because people route around it. Four practices keep it fast:
- 1.Approve in batches where the risk allows. Twenty purchase-order drafts reviewed together take less time than twenty interruptions.
- 2.Approve by exception for low-risk internal actions, with sampling, and keep sign-off on every external action.
- 3.Measure approval latency as a KPI from the first week. If drafts wait a day, the bottleneck is the approver's workload, and that is a staffing decision, not an AI one.
- 4.Train the approvers before launch. By the time an agent is live, the people in the loop have corrected its drafts for weeks and know what to look for.
The share of actions that need a person should fall as the agent proves itself on internal work. It should never fall to zero for external actions. That line is what makes the agent governable and the firm accountable.
Map it to the frameworks your examiners use
- ◆NIST AI RMF: the Govern, Map, Measure, and Manage functions. The approval map and the record are most of Govern and Manage.
- ◆Federal Reserve SR 11-7 (model risk management): inventory, documentation of purpose and limits, validation, and ongoing monitoring. An agent that drafts filings belongs in the model inventory with that documentation.
- ◆EU AI Act, Article 14: human oversight for high-risk systems, including the ability to understand, intervene, and stop. The gate is how you demonstrate it.
- ◆HIPAA Security Rule, 45 CFR 164.312: access controls and audit controls. Credential-gated clinical functions and a record of every clinical action meet both.
How KOI builds it in
Every KOI deployment carries the three elements: a policy check before any consequential action, a named human sign-off on everything client-facing, and an accountability ledger the client can audit. The design behind that governance and traceability is KOI's own and is patent pending. What a client sees is simpler: drafts to approve in the systems they already use, a record that answers the examiner's question, and an engagement in which one senior consultant signs off every step.