“A human reviews it” is the most frequently claimed and least examined control in AI governance. It becomes real only when the organisation has decided which actions require review, what the reviewer must verify, what competence and authority they hold, and what happens when they say no.
The four-part test. Oversight is a control only when the reviewer can see enough to disagree, has enough time to disagree, is competent to disagree, and disagreement changes what happens. Remove any one of the four and what remains is a confirmation step that manufactures evidence of nothing.
Control design reference · EU AI Act, Regulation (EU) 2024/1689 · ISO/IEC 42001:2023 · Regulatory and vendor terms change — verify current sources · Last reviewed August 2026
Human oversight becomes a control at the moment four things are true together: the organisation has written down which specific actions require review rather than asserting that outputs are reviewed, the reviewer has been told what to check and can check it against something the model did not produce, that reviewer has the competence and the authority to reject, and rejection has a defined path that is actually used. A programme missing any one of those has a workflow step rather than a control — and at an assessment the two are distinguished by records, not by intent. None of this is about the quality of the model. It is about the design of the gate, the arithmetic of the queue, and whether the record can show that a person could have said no and did not. Where most programmes fail is the last part: a policy stating that humans review AI outputs is a statement of intent, and the evidence is a population of reviewed items showing what each reviewer was shown, who they were, when they decided relative to the action, what they decided, and what happened to the ones they rejected. This page is the companion to the sixth of the ten questions that govern any AI agent, and it goes where that page could only gesture.
Key Takeaways
- Oversight is a control only when the reviewer can see enough to disagree, has enough time to disagree, is competent to disagree, and disagreement changes the outcome. Remove any one of the four and what remains is a confirmation step that produces evidence of nothing.
- Uniform review is the most common design error. Routing everything to a gate guarantees that the items which genuinely needed attention are cleared at the same speed as the ones that did not, so review depth has to be matched to consequence rather than applied evenly across a population.
- The two structural failure modes are volume and deference. Rubber-stamping is arithmetic — a queue demanding more decisions per hour than reading the items allows will be cleared at reading speed — while automation bias is psychological, and the research on automated decision aids reports both omission and commission errors in expert and inexperienced reviewers alike.
- A gate placed after the consequential act has already occurred is a quality process rather than an oversight control: a sent message, a released payment and a deployed change are outside your control the moment they land, and reviewing them afterwards changes what you learn rather than what happened.
- Refusal has to be cheap and consequential at the same time. A rejection path that costs the reviewer an argument, a ticket and a week is a path nobody takes, and a rejection that returns the same item to the same queue unchanged is not a refusal at all.
- A policy saying humans review is not evidence that they did. The record has to show, per item, what the reviewer was shown, who they were, when they decided relative to the action, what they decided, and — for the rejections — what happened next.
What “a human reviews it” has to be replaced with
The sentence is not wrong so much as empty. It contains no action list, no verification instruction, no competence requirement and no failure path, which means it can be true of a control that works and equally true of one that has never caught anything. Four answers replace it, and each one closes off a way the next could be answered vaguely. Read “An answer that holds” on each card as the standard, and “The answer that fails” as the version that is currently in most policy libraries.
Which actions require review?
Name the actions by verb and by threshold, not by system. Review attaches to consequence — external, irreversible, financial, privilege-changing, or affecting an individual’s rights — and everything below that line is either sampled after the fact or deliberately not reviewed. Write the rule so a person outside the team could sort yesterday’s traffic into gated and ungated and reach the same answer you would.
An answer that holds
Outbound messages to a counterparty that assert an amount owed, plus any message to an account carrying a dispute flag. About six per cent of traffic, defined by two fields the system already holds.
The answer that fails
“All AI-generated content is reviewed before use.” It covers everything, so it prioritises nothing, and it is contradicted by the first busy Tuesday.
What must the reviewer actually check?
A short list of the specific things that can be wrong with this output, in the order that matters, written by whoever knows the failure modes. The reviewer’s job is verification against a source, not a judgement of plausibility — and plausibility is precisely what a fluent model is optimised to produce. A check that can only be performed by reading the output is not a check.
An answer that holds
Three checks: the figure reconciles to the ledger balance as at today, the account is not on an active payment plan, and the disputed-account tone rule is applied. Each is verifiable against something the model did not write.
The answer that fails
“Review for accuracy and appropriateness.” Nothing in that sentence tells a reviewer what to open, so the review collapses into reading the draft and finding it reasonable.
What competence does the reviewer need?
Enough understanding of what this system does well and badly to know where to look, enough domain knowledge to catch a wrong answer that reads correctly, and enough standing that a rejection is not a career event. The EU AI Act names the three parts for deployers of high-risk systems — competence, training and authority — and the third is the one organisations quietly skip.
An answer that holds
A senior collections analyst who has seen the model’s failure pattern on partial payments, has worked through the twelve real failures in the training pack, and can stop a message without asking permission.
The answer that fails
Whoever is on the rota. Frequently the most junior person available, because from the outside reviewing looks like a low-skill task.
What happens on disagreement, and on failure?
Define the rejection path before go-live: where a rejected item goes, who owns fixing it, whether the same output may be resubmitted unchanged, who may override a refusal and on what record, and what a reviewer does when they are unsure rather than opposed. Then define the failure path — what happens when the gate is bypassed, when nobody reviews inside the window, and when the queue exceeds capacity.
An answer that holds
Rejections carry a reason code, route to a named account owner, and cannot be resubmitted without a change. Overrides require the accountable owner, preserve the original refusal, and are counted monthly.
The answer that fails
The reject button exists and nothing downstream is defined, so the item either vanishes or returns identical. Reviewers learn within a fortnight that approving is the only action with a predictable result.
The four map onto the law more closely than they might appear to. For deployers of high-risk systems, Article 26(2) of the EU AI Act — Regulation (EU) 2024/1689 — requires that human oversight be assigned to natural persons who have the necessary competence, training and authority, as well as the necessary support. That single sentence carries question 3 entirely and most of question 4, because authority without a usable refusal path is a word on an org chart. Under ISO/IEC 42001:2023 the same ground is covered by the responsible-use controls in Annex A.9 together with competence under clause 7.2, and by whatever your AI risk and impact assessments concluded about where a person needs to stand between the system and the world.
Eight ways a gate stops working without anyone noticing
None of these announce themselves. In every case the control continues to produce approvals, the dashboard stays green, and the policy remains accurate as written. The third column is what to look for in your own data, and it is worth noting that seven of the eight are visible in an export you can produce this afternoon.
| Failure mode | What is actually happening | Signature in the data | Design response |
|---|---|---|---|
| Rubber-stamping under volume | Arithmetic, not attitude. A queue holding more items than the available attention gets cleared at whatever speed empties it, and the reviewer is behaving rationally given the target they were set. | Median time-on-task below the time it takes to read the item. Approval rates above 99 per cent sustained for months. Decisions clustered in bursts at the start and end of the day. | Narrow what is gated until the remainder fits the available attention, and move the rest to sampling. Adding reviewers scales cost linearly and leaves the incentive intact. |
| Automation bias | Deference to the machine’s output as a substitute for independent checking. The human-factors literature on automated decision aids distinguishes omission errors, where the reviewer misses something the system did not flag, from commission errors, where the reviewer follows an incorrect recommendation despite contrary information being available. Both have been observed in expert as well as inexperienced participants. | Rejections concentrated almost entirely in items the system itself flagged as uncertain, and close to none in items it presented confidently. | Make the reviewer accountable for the decision rather than for clearing the queue, and establish that before they decide — accountability imposed in advance has been found to reduce both error types. Where the decision warrants it, capture the reviewer’s own reading before revealing the recommendation. |
| Reviewing the machine’s account of itself | The reviewer is shown a summary, a confidence score or a rationale produced by the same system whose output is under review. The check inherits the error it was meant to catch. | A review surface with no link to any source system. Reviewers who cannot say, when asked, what they open before deciding. | Every item on the checklist must be verifiable against something the model did not generate. If it cannot be, it is not a check — it is a second reading. |
| Context poverty | The reviewer sees the output and not the situation: no account state, no history, no record of what the system was asked or what it retrieved. | Rejections reversed on appeal at a high rate, and reviewers who describe the work as guessing. | Put the three or four facts the checklist requires into the review surface itself. A review that needs four browser tabs will be performed in one. |
| Authority deficit | The reviewer can flag but not stop, or can stop only at a personal cost — an argument with a product owner, a delayed release, an escalation carrying their name. | A rejection rate near zero alongside a high volume of comments, side-channel messages, or approvals annotated with concerns. | Give refusal a formal, low-friction record, and put the override — not the refusal — on the accountable owner’s desk. Then count overrides. |
| The gate sits after the act | Review placed downstream of the moment the action became irreversible. The message has sent, the payment has left the account, the change is in production. | Approval timestamps later than action timestamps in the same record, or a review cadence measured in days against an action that completes in seconds. | Move the gate, or accept that the control is detective and describe it as such — including in the customer questionnaire, where calling a post-hoc quality review a human-in-the-loop control is the claim least likely to survive a competent buyer. |
| The reviewer is the requester | The person who asked for the output approves it. That is not oversight. It is self-review with an extra click, and it collapses a segregation of duties that existed before the system arrived. | Approver identity equal to initiator identity on a material share of items, or an approver that is the agent’s own service identity. | Enforce non-self-approval in configuration rather than in policy, and record the prohibition explicitly. It is among the first things a buyer’s security team tests. |
| Sampling that is not sampling | “We spot-check” with no defined rate, no defined population and no record of which items were checked — usually chosen by the person who produced the work. | No exportable list of sampled items. A defect rate that cannot be calculated because the denominator was never fixed. | Define the population, the rate, the selection method and the reviewer, and record every sampled item including the ones with no finding. An unrecorded clean sample is the half of the evidence people throw away. |
The second row deserves its own paragraph because it is the one people dismiss. Automation bias is not carelessness and it does not respond to being told about it. The classical studies of automated decision aids in aviation and clinical settings found that participants made omission errors, failing to act on a problem the automation did not flag, and commission errors, acting on an incorrect automated recommendation in the face of contradictory information that was available to them — and found this in experienced professionals as readily as in novices. The finding that matters most for control design is a hopeful one: imposing accountability for the accuracy of the decision, established before the decision rather than after it, measurably reduced both error types. That is a design instruction. Reviewers who understand that they own the outcome, and who are measured on decision quality rather than on queue throughput, behave differently from reviewers who are processing a list.
The broader research literature is not encouraging about human oversight as a general remedy. A 2022 survey in Computer Law & Security Review examined forty-one policies requiring human oversight of government algorithms and argued that they suffer from two connected flaws: people frequently cannot perform the oversight function the policy assumes, and the policy then legitimises deployment of the system without addressing the underlying problem. The conclusion to draw is not that oversight is worthless. It is that oversight asserted at the level of a policy sentence is worthless, and that the burden falls on whoever designs the gate to show that the specific people, on the specific queue, with the specific information in front of them, can actually do the job the sentence claims.
Where the review sits decides what it can prevent
The most consequential decision in oversight design is not how thorough the review is but where in the workflow it sits, because that alone determines whether the control is preventive or detective. The deciding variable is reversibility: find the moment at which the action stops being yours to undo, and put the gate before it. An email that has been delivered, a payment that has left the account and a change that is running in production are all past that moment, and a review afterwards changes what you know rather than what happened.
| Where the gate sits | What it can still prevent | What it costs | When to choose it |
|---|---|---|---|
| Before generation — the request is gated | The output from existing at all, including the data exposure involved in producing it. The only placement that can prevent retrieval of something the requester should not have seen. | Latency on every request, and a queue that sees overwhelmingly legitimate traffic. | Where the risk sits in the asking: sensitive data classes, regulated populations, or a use case that has not been approved. |
| Between generation and action — the classic gate | The consequential act. The output exists and can be examined, and nothing external has yet happened. | Latency proportional to review depth, with throughput hard-bounded by reviewer capacity. | The default for anything external, irreversible, financial or rights-affecting. |
| Staged — the system proposes, a human releases in batches | The same acts, at lower unit cost, because the reviewer works a set rather than an interruption and can see patterns across items. | Delay bounded by the batch interval. Unsuitable where the action is time-critical. | High-volume, low-variance work where the reviewer’s value lies in detecting an anomalous item among many rather than in depth on one. |
| In-flight — thresholds and circuit breakers | Escalation of a failure already under way. It does not prevent the first bad action, only the ninetieth. | Design effort, plus false stops that erode confidence in the mechanism. | Unattended runs where no human is present to gate anything — as a complement to a pre-action gate, not a replacement for one. |
| After the act, every item — post-hoc review | Nothing. It detects, and it feeds correction and tuning, which are worth having but are not the same claim. | The full review cost with none of the prevention, which makes it the most expensive honest option on this list. | Almost never as a primary control. If you are already paying to look at every item, look before the act instead. |
| After the act, sampled — assurance | Nothing directly. It is the control that tells you whether the tiers you chose not to gate are behaving as assumed. | Low, and predictable, because it scales with the sample rather than with volume. | Everywhere something is deliberately ungated. Sampling is the price of a narrow gate. Without it, narrowing the gate is simply removing the control. |
Two placements get confused constantly, and the confusion is expensive because it usually surfaces in a customer questionnaire rather than internally. Post-hoc review of every item feels like the most rigorous option — you are, after all, looking at everything — while being the only option on the list that prevents nothing at all. If the budget exists to examine every item, move that budget upstream. And where a genuine constraint keeps the gate downstream, such as an unattended overnight run, say so and pair it with in-flight ceilings and circuit breakers so that the first bad action cannot become the hundredth. Describing a detective control accurately is not a weakness in a security review. Describing it as a human-in-the-loop gate is the claim that does not survive the follow-up question.
Review design by action, not by system
Oversight attaches to verbs. The same product, the same model and the same team warrant completely different gates depending on which action is being taken, which is why a review policy written per system is always either too heavy or too light and usually both at once. This table is the working version of the review-design decision: nine action classes, what the gate should be, what the reviewer is actually verifying, a realistic time budget, who is competent to hold it, and what refusal does. The time column is a planning range from readiness work, not a benchmark.
| Action class | Gate | What the reviewer verifies | Time budget | Who is competent | What refusal does |
|---|---|---|---|---|---|
| Read and retrieve | None per item. Control the corpus and the permissions instead. | Nothing per item. The reviewable object is the retrieval scope, examined once at approval and again on change. | — | Data owner, at approval time. | Refusal happens at the permission layer, not in a queue. |
| Internal write — draft, ticket, record in a non-final state | None per item, sampled after the fact. | In the sample: whether the write is attributable to the agent identity, reversible, and within the scope that was approved. | 3–5 min per sampled item | Process owner for the affected record type. | Reverse the write and log a defect. There is no live decision to make. |
| Outbound message to a person outside the organisation | Pre-send for the defined subset, sampled for the remainder. | Every factual claim against a source system, the recipient and the account state, and whether the message commits the organisation to anything it has not agreed. | 2–4 min | Someone who owns the relationship, not a generalist on a rota. | Reason code, route to the named account owner, no resubmission without a change to the item. |
| Financial transaction or payment release | Pre-execution above a stated value, always. Velocity and aggregate ceilings below it. | Payee against master data and against any bank-detail change in the last thirty days, amount against the source document, and that the batch total reconciles to the underlying items. | 5–15 min per batch | Whoever holds the payment authority under the existing scheme of delegation. | The batch is held whole, never partially released. Override requires the accountable owner and a written reason. |
| Deployment or change to production | Pre-merge approval by a non-author, enforced by branch protection rather than by convention. | The diff, the test result, the blast radius, and that the change matches an intent someone approved before it was written. | 10–40 min | An engineer with ownership of the affected service. | The pull request stays open. Nobody may bypass, and the agent identity is excluded from the bypass list. |
| Deletion or destructive change | Pre-execution, with soft delete and a tested recovery window as the design default. | What is being removed, what depends on it, whether it falls under a retention obligation, and that the recovery path has actually been exercised. | 5–20 min | System owner, plus the owner of any retention obligation the record falls under. | Deletion does not proceed. The item stays soft-deleted until the question is settled. |
| Approval of another item — clearing a request, closing a finding, signing off a review | Do not grant this verb to an AI system. Where it is granted, the item is reviewed in full by a human holding the underlying authority. | Everything the human approver would have verified. There is no shortcut, which is the point. | The same as the original human process | Whoever held the approval before it was automated. | The item returns to the human queue and stays there. |
| Privilege or configuration change — permissions, roles, other agents, the gate itself | Pre-execution by someone other than the requester, recorded outside the system being changed. | What the change grants, to whom, for how long, and whether it alters the oversight configuration in any way. | 10–30 min | Security or platform owner. | No change. Any change to gate configuration escalates to the accountable owner by default rather than on request. |
| Decision affecting an individual — hiring, credit, eligibility, benefits, discipline | Pre-decision, non-negotiable, by a person able to reach a different outcome on the merits. | The inputs, the individual’s own submissions, whether the recommendation is being followed for a reason or by default, and that the challenge route is available and explained. | As long as the decision deserves | The decision-maker in the underlying process, trained on this system’s known limitations. | The human decision is the decision, and is recorded as such. The recommendation is an input, not an outcome. |
Three rows carry most of the argument. The approve row is the one to hold the line on, because granting an AI system the ability to clear another item collapses a segregation of duties without altering a single documented process — the approvals still happen, the record still shows them, and the second pair of eyes has quietly become the first pair again. The privilege and configuration row is the one that includes the gate itself, and a gate whose thresholds can be moved by the team whose work it slows down is a gate with a scheduled expiry. And the decision affecting an individual row is where the standard is highest and the law is most specific.
On that last row: where an output produces legal effects or similarly significantly affects a person, European data protection guidance has been explicit for years that oversight must be meaningful rather than a token gesture, and must be carried out by someone with the authority and competence to change the decision — with employees rubber-stamping automatically generated profiles named specifically as insufficient. The Court of Justice reinforced the reach of that analysis in the SCHUFA case (C-634/21, 7 December 2023), holding that automated scoring can itself constitute an automated decision where the recipient of the score draws strongly on it. The operational consequence is precise: it is not enough that a person clicked. The record has to show that they had the information and the standing to reach a different outcome, and the only way to show that later is to have captured it at the time.
Two rows have their own deeper treatments elsewhere: financial actions in spending and transaction limits, and deployment in coding agents and production access. Both make the same underlying point from different directions — that a ceiling enforced in configuration does work no reviewer can do, and that the reviewer should be spending their attention on the items the ceiling could not classify.
Match the review to the consequence, not to the population
Uniform review is a decision to spend the same attention on an internal draft as on a payment release, and since attention is finite the result is that both receive the attention appropriate to the draft. Five tiers are enough for almost every programme. The rule that makes them work is that an item moves up a tier by acquiring a consequence, never by acquiring visibility — and that every tier below the gate is covered by the tier above it in the form of sampling.
| Tier | What qualifies | Depth of review | Throughput | Evidence produced |
|---|---|---|---|---|
| Tier 0 · Not gated | Reversible, internal, no individual affected, no external commitment, no privilege change. | None per item. | Unbounded. | The scoping rule itself, and the sampling result from Tier 1 that shows the tier is behaving as assumed. |
| Tier 1 · Sampled assurance | Reversible or internal, but high volume, or feeding a downstream process that trusts the output. | The full checklist, applied to a defined sample at a stated rate. | Bounded by sample size rather than by volume, so it survives growth. | The sample list, the selection method, the completed checklist per sampled item, and the defect rate trended over time. |
| Tier 2 · Targeted check | Externally visible but bounded, or financial below a stated threshold. | Named checks only — three or four, each verifiable against a source outside the output. | Roughly 15–25 items per reviewer-hour at three minutes each, before fatigue is taken into account. | Per item: what was shown, who decided, when relative to the action, and what they decided. |
| Tier 3 · Full review | Irreversible, financial above threshold, privilege-changing, or affecting an individual’s rights or access. | The complete decision on the merits, with the system’s recommendation treated as one input. | Low by design. Where the volume does not fit, the answer is less automation rather than a faster gate. | Everything in Tier 2, plus the reviewer’s own reasoning wherever they departed from the recommendation. |
| Tier 4 · Dual control | Where a single reviewer’s error is unrecoverable, or where the law requires two people. The EU AI Act requires separate verification and confirmation by at least two persons with the necessary competence, training and authority before acting on certain biometric identifications. | Two independent reviewers, neither able to see the other’s decision first. | Very low. Reserve it for the cases that genuinely warrant it. | Two decision records with independent timestamps, and evidence that the second reviewer could not observe the first. |
Do the arithmetic before the design is approved. A seven-hour working day at fifty per cent review occupancy — which is generous, since most reviewers hold the rota alongside their own caseload — gives 210 reviewer-minutes a day. At three minutes an item that is seventy items per reviewer-day. Two reviewers therefore support a gated population of about 140 items a day, and a design that routes 300 items a day to those two people is arithmetically broken before the first item arrives. This calculation takes five minutes, needs no data you do not already have, and is the single most useful thing a governance function can do at the point a gate is being designed rather than six months later when the queue export tells the same story in a more expensive way.
Tier 4 exists partly because the law occasionally requires it. Where a deployer acts on a post-remote biometric identification, the EU AI Act requires that no action be taken unless the identification has been separately verified and confirmed by at least two natural persons with the necessary competence, training and authority — with a carve-out for certain law-enforcement, migration, border-control and asylum uses where Union or national law considers the requirement disproportionate. That is a narrow provision, but the structure it describes — two people, independently, before the act — is the right pattern for any decision where a single reviewer’s error cannot be undone, regardless of whether the Act reaches you.
Five capabilities the reviewer needs, and how you show they have them
Article 14(4) of the EU AI Act sets out what the people assigned to oversee a high-risk system must be enabled to do. It is the most useful short specification of reviewer capability in circulation, because each item was written against an observed failure rather than an abstraction — and because it forces the question every programme skips, which is how you would demonstrate any of it to somebody who was not there. The last column is where oversight programmes are lost.
| What the reviewer must be able to do | What that means operationally | How you establish it | How you evidence it |
|---|---|---|---|
| Understand the system’s capacities and limitations, monitor its operation, and detect anomalies, dysfunctions and unexpected performance — Article 14(4)(a) | The reviewer can state, in their own words and without prompting, what this system gets wrong and roughly how often. | A training pack built from your own failures — the actual wrong outputs with the account context, not a generic page about model limitations. | Dated training records naming the system and the version reviewed, plus the pack itself with a revision history. |
| Remain aware of the tendency to rely or over-rely on the output — automation bias — Article 14(4)(b) | The reviewer treats the recommendation as one input and knows they are predisposed not to. | Name the bias explicitly in training, measure the rejection rate, and show reviewers their own numbers rather than the team average. | The metric itself, reviewed on a stated cadence against a threshold, and a record of what was done when it moved. |
| Correctly interpret the output, taking into account the interpretation tools and methods available — Article 14(4)(c) | The reviewer knows what a confidence score does and does not mean, and does not read fluency as correctness. | Document what each element of the review surface represents and what it is not evidence of. Confidence is not accuracy and a citation is not a verification. | The reviewer-facing documentation, versioned, plus a competence check with recorded results per reviewer. |
| Decide not to use the output, or otherwise disregard, override or reverse it — Article 14(4)(d) | The path exists, it is used, and using it has a defined and survivable consequence for the person who used it. | Build the rejection path first and the approval path second. | Rejections present in the population, with reason codes and downstream outcomes. A period containing none is a question you must be able to answer. |
| Intervene or interrupt the system, including through a stop mechanism that brings it to a halt in a safe state — Article 14(4)(e) | Someone on shift can stop it without raising a vendor ticket, and knows what state the stop leaves behind. | Design the stop layer by layer, then test it on a date and keep the record. | A dated stop test with elapsed times and an honest note on what was still in flight afterwards. |
A word on where these obligations currently bite, because the timing has moved. The AI Act is Regulation (EU) 2024/1689. It was amended by the Digital Omnibus on AI, Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026 and in force from 27 July 2026, which deferred the application of obligations for stand-alone high-risk systems under Annex III — employment, education, credit assessment, law enforcement, critical infrastructure and similar — from 2 August 2026 to 2 December 2027, and for AI embedded in products already covered by Union product-safety law under Annex I to 2 August 2028. Two things follow. The Article 14 and Article 26 duties still arrive, with a longer runway than many programmes were planning against. And the deferral changes nothing about the five capabilities as a design standard, which is why the useful posture is to build to them now and treat the date as a compliance milestone rather than as a start signal. The relationship between the Act and the certifiable management system is set out in ISO 42001 versus the EU AI Act.
Under ISO/IEC 42001:2023 the same requirements arrive through management-system machinery rather than through a capability list. Clause 7.2 requires competence to be determined, provided and evidenced for people whose work affects performance. Clause 7.3 requires awareness, and clause 5.3 and the roles control in Annex A.3 put a name against the responsibility. Annex A.9 carries the use-side requirements — processes for responsible use, objectives for responsible use, and intended use — and the operation and monitoring control in Annex A.6 is where a running gate lives. Nothing in the standard prescribes a gate design, which is the point: it requires you to have decided, and to be able to show what you decided and that it operated. The organisational-capability question sits alongside this in AI competence and training obligations.
Making no an available answer
The approval path gets designed carefully because it is the path everyone expects to use. The rejection path gets a button. Then the rejection rate comes in at nought point one per cent and everybody concludes the model is performing well, when what has happened is that refusing costs the reviewer more than approving and they have priced it correctly. Six design rules make refusal a live option rather than a theoretical one, and none of them are expensive if they are built before go-live.
A rejected item has to go somewhere named
The most common reason a refusal path is never used is that nobody defined what happens after the button. The rejected item needs a named owner, a queue that is monitored, and a resolution state — corrected and released, escalated, or abandoned with a reason. If the destination is “back to the model”, the reviewer has learned that rejecting produces the same item again with a delay, and they will stop. Write the destination into the control description, because an assessor selecting a rejection will ask what happened next, and the absence of a successor record turns your best evidence of a working gate into evidence of a dead end.
Resubmission without change is not permitted
If an item can be re-queued unchanged, refusal is advisory. The rule is mechanical: a rejected item requires a recorded change — to the input, the parameters, the context or the human decision — before it re-enters the queue, and the new item carries a reference to the refusal it descends from. This costs almost nothing to build at design time and is close to impossible to retrofit, because by then the queue holds thousands of items with no lineage. It also produces the single most persuasive artefact in a customer security review: a chain showing a refusal, a change, and a different outcome.
Override is a different act from approval, and it is counted
Sometimes a refusal has to be reversed, and pretending otherwise produces workarounds instead of records. Make override a distinct act with a distinct approver — the accountable owner, not the requester — that preserves the original refusal in the record rather than replacing it. Then count overrides monthly and read the reasons. A refusal path that is always overridden teaches reviewers within weeks that refusing costs them an argument and changes nothing, which is exactly the state you were trying to avoid. A refusal path with no overrides at all is worth a look too: it may mean nobody is refusing.
“Not sure” needs its own button
Reviewers are frequently neither satisfied nor opposed. They are uncertain, and a two-value interface forces uncertainty to resolve as approval because approval is the path with no friction. A third option — escalate, hold, or ask — costs one extra state and changes the distribution of outcomes measurably. It also produces a signal worth having: items that reviewers cannot resolve cluster around the system’s genuine weak points far more reliably than rejections do, because a rejection requires the reviewer to already know what is wrong.
The reviewer’s targets must not punish refusal
If reviewers are measured on queue throughput, resolution time, or a service level that a rejection breaches, the incentive is settled and no amount of policy language will move it. Measure reviewers on decision quality assessed by a periodic second look, not on items cleared per hour. Where a business service level genuinely applies, exclude rejected items from it explicitly and say so in the reviewer’s own documentation, because reviewers price the consequences of refusing far more accurately than governance teams expect.
The gate’s own configuration is inside the control
A gate that can be reconfigured without approval is a gate for as long as it is convenient. Thresholds, the gating rule, the checklist and the review surface all belong under change control, with the change record held outside the system being changed and with the accountable owner as approver. The specific failure to guard against is the mid-period threshold move: an assessor examining a period will notice that the value gated at ten thousand in January was gated at fifty thousand by June, and the question is not whether the change was reasonable but whether it went through anything.
There is a test for whether any of this is real, and it takes one question: when did a reviewer last stop something, and what happened next? If nobody can name an instance, the refusal path has never been exercised, and an untested path is indistinguishable from an absent one — the same reasoning that makes an untested restore no backup and an untested stop no kill switch. Where the honest answer is that nothing has ever been rejected, the next question is whether the population is genuinely that clean, and the way to find out is the sampled defect rate on the items nobody gated.
Twelve fields that turn a review into evidence
This is where most oversight programmes fail an assessment, and it is rarely because the control was not operating. It is because the control operated and left a record consisting of the word approved. The twelve fields below are what a review record has to carry for the control to be testable by somebody who was not in the room. Build them at design time: eight of the twelve cannot be reconstructed afterwards at any price, and the four that can will cost more to reconstruct than they would have cost to capture.
| Field | Why it is load-bearing | Where it comes from | What its absence means |
|---|---|---|---|
| Item identifier, plus the identifier in the downstream system | Ties the review to the thing that actually happened. Without the second identifier you can prove a review occurred and not that it was a review of this action. | The workflow platform, plus the reference the target system accepted and stored. | The review population cannot be reconciled to the action population, so neither one is evidence of the other. |
| What the reviewer was shown | The single field that separates a record from a flag. If the review surface changed mid-period, the earlier decisions were made about something else. | A stored snapshot per item where volume allows, or a versioned definition of the review surface with change dates where it does not. | An approval with no visible basis. Routinely rejected, and correctly so. |
| Reviewer identity, as a person | Accountability does not survive a shared account, and self-approval is invisible without it. | The identity provider, resolved at decision time rather than at export time. | A shared queue account, a role name, or the agent’s own service identity — which is self-approval wearing a badge. |
| Decision timestamp and action timestamp | Ordering is the entire preventive claim. An approval recorded after the action evidences a quality review, not a gate. | Both systems, normalised to one time zone with the offset recorded. | One timestamp only, or two in different zones with no offset — which an assessor will treat as unresolvable rather than as favourable. |
| Time on task | The only field that tests whether the review was possible, as distinct from recorded. It is also the first thing an experienced assessor computes. | Item-opened to decision-recorded, captured by the workflow rather than inferred later. | You cannot answer the opening question of the conversation, and the burden shifts to you to show the review was real. |
| The decision, with at least three values | Approve, modify and reject are different outcomes, and modification is where most reviewer value actually appears. A two-value control cannot show it. | The workflow, with modification recorded as its own state rather than as an approval. | Approve-only workflows report a hundred per cent approval rate by construction, and then have to explain it. |
| Reason code on anything other than a clean approval | Turns the rejection population into a diagnostic instead of an anecdote, and makes tuning arguments resolvable by data. | A short controlled list maintained by the process owner, with free text as a supplement. | A rejection rate you can quote and cannot explain, which is weaker than a lower rate you can. |
| What changed, where the decision was modify | The difference between the proposed output and the released one is the clearest available proof that a human engaged with the content. | A stored diff, or the released version alongside the proposed version. | Modification collapses into approval in the export, and the strongest evidence you had disappears. |
| Downstream outcome of a rejection | A refusal is a control only if something happened next. Otherwise it is a click. | The remediation ticket, the owner’s record, or the successor item with its lineage reference. | Rejections that end at the reject button, which reads as a gate nobody is behind. |
| Override record where a refusal was reversed | Overrides are legitimate and their absence from the record is not. The original refusal has to remain visible alongside the reversal. | The workflow, with the accountable owner as approver and a written reason. | An override log that replaces rather than preserves the refusal, which makes the refusal population unreliable. |
| Reviewer competence as at the date of the review | Training records are dated. A reviewer added to the rota in month four needs a record in month four, not a record from the original rollout. | The learning system or the training register, joined to the reviewer population. | A competence claim covering the people you remembered, tested against a population containing several you did not. |
| Population definition and export metadata | The export is information produced by the entity. The query, the filter, the date range and the record count are what make it testable. | Captured at export time, on the face of the file. | A review log that is an assertion rather than evidence, however complete it happens to be. |
Two of the twelve are worth building even if you build nothing else. Time on task is the field that makes every other claim about the gate credible, and it is a single subtraction that no workflow platform will capture unless you ask it to. What the reviewer was shown is the field that converts a decision into a decision about something — and it is the one most often lost, because review surfaces are iterated continuously and nobody versions them. Where storing a snapshot per item is impractical, version the surface definition and record the change dates, so that a decision from March can be placed against the March interface rather than against today’s.
The same split that governs agent logs applies here. Keep the decision skeleton — the twelve fields, minus the content — separate from the payload, because the skeleton is small, contains little personal data, and needs to survive for the whole assessment period, while the payload is large, frequently contains customer personal data, and should be governed by minimisation. Holding both in one store with one retention rule forces a choice between evidencing the control and minimising the data, and that choice usually gets made by whoever set the default. The parallel treatment for agent activity is in monitoring agents that work unattended.
Eleven rejections in a quarter, and what was wrong with that
An illustrative composite, not a client. A 260-person B2B payments company runs collections and billing through a platform whose GenAI feature drafts customer messages from account state. A workflow step routes every draft to a human before it sends. On paper the control is textbook. Read the two rows dated 14 April first.
| Design | Nov 2025 | The control is written as: all AI-drafted customer messages are reviewed and approved by a member of the collections team before sending. It is approved on that wording, and the wording is the whole problem — it names no checks, no threshold, and no capacity. |
| Live | 5 Jan 2026 | The gate goes live across the entire queue. Two analysts hold the rota alongside their own caseloads. |
| Quarter one | 1 Jan – 31 Mar | 18,400 drafts routed across 62 working days: 297 a day against a two-reviewer capacity that allows 85 seconds an item at full occupancy. Eleven rejections. Approval rate 99.94 per cent. |
| The number | 14 Apr | A readiness review exports the queue. Median time-on-task is 9 seconds. Reading the median 148-word draft takes about 37 seconds at ordinary reading speed, and a timed walk-through of the three checks the team believed it was performing takes 2 minutes 51 seconds. Sixty-one per cent of decisions were recorded within five seconds of the item opening. |
| The second number | 14 Apr | Reviewer concentration: 13,064 of 18,400 decisions — 71 per cent — were taken by one of the two analysts, and 2,190 of those fell on the eleven days when the other was on leave. On those days one person cleared an average of 199 items alongside a full caseload. |
| Redesign | 22 Apr | The gate is narrowed rather than reinforced. Two fields the system already holds define the gated subset: the message asserts an amount owed, or the account carries a dispute flag. Everything else moves to sampled assurance, plus an automated rule that blocks any message whose figure does not reconcile to the ledger. |
| Checklist | 24 Apr | Three named checks replace “accuracy and appropriateness”: the figure reconciles to the ledger balance as at today, the account is not on an active payment plan, and the disputed-account tone rule is applied. Each is verifiable against something the model did not write, and the review surface is rebuilt to show all three without leaving the screen. |
| Live again | 1 May | The narrow gate goes live with a rejection path: reason code, route to the named account owner, no resubmission without a change, override only by the collections manager and counted monthly. The rota widens from two analysts to four account owners, because the gated item now belongs to somebody specific. |
| Quarter two | 1 May – 31 Jul | Comparable volume. 1,104 items gated — 6.0 per cent — across 64 working days, 17 a day, four a day per reviewer. Median time-on-task 3 minutes 4 seconds. Median queue latency 41 minutes, 95th percentile 3 hours 10 minutes. Ninety-six rejections: 8.7 per cent. |
| What it caught | 1 May – 31 Jul | Of the 96 rejections: fourteen figures that did not reconcile, three notices addressed to accounts already on a payment plan, and one message to an account whose dispute had been raised that morning. Eleven overrides, all by the collections manager, all recorded with a reason and with the original refusal preserved. |
| Sampling | weekly | Fifteen ungated messages a week reviewed against the same checklist — 195 over the quarter — producing a defect rate of 1.5 per cent, three items, none of them externally consequential. That figure is the evidence that narrowing the gate was safe rather than merely cheaper. |
| Assessment | 11 Aug | The request list asks for the population of gated actions and the reviews performed. Both quarters export cleanly. The quarter-one file goes in with a note explaining what the numbers showed and what changed on 1 May — which is a stronger artefact than a clean quarter would have been. |
Nothing was wrong with the eleven rejections. What was wrong was the 18,389 approvals that nobody had time to make.
The arithmetic is the part worth carrying away, because it is the same in every organisation and it is always available before the fact. Two reviewers at fifty per cent occupancy in a seven-hour day give 420 reviewer-minutes. Spread across 297 items that is 85 seconds an item — and the three checks the team believed it was performing take 2 minutes 51 seconds when timed, of which 37 seconds is simply reading the draft. The gate was impossible on the day it was designed, by a factor of two, and the nine-second median was not a failure of diligence. It was two competent people responding rationally to a queue that could not be done properly and had to be emptied.
| Measure | Q1 · gate over everything | Q2 · gate over 6 per cent | What it tells you |
|---|---|---|---|
| Items gated | 18,400 (100%) | 1,104 (6.0%) | The single design decision on the page. Everything else follows from it. |
| Items per reviewer-day | 149 (two reviewers) | 4 (four reviewers) | Capacity at three minutes an item is roughly 70 a day per reviewer at realistic occupancy. Quarter one was over capacity by more than a factor of two before the first item arrived. |
| Median time on task | 9 seconds | 3 min 04 s | Against 37 seconds to read the median draft and 2 min 51 s to perform the three checks, timed. |
| Rejection rate | 0.06% (11 items) | 8.7% (96 items) | Not a target and trivially gameable. What makes the second figure meaningful is that each rejection carries a reason code and a downstream outcome. |
| Reviewer concentration | 71% on one of two | 39% on one of four | A continuity risk that appears in no policy document and in every queue export. |
| Median review latency | Not captured | 41 min · P95 3 h 10 m | An uncaptured latency is a bypass request waiting for a justification. |
| Reviewer hours in the quarter | 46 | 69 (56 gate + 13 sampling) | About 1.5 times the cost. |
| Findings | 11 | 96, plus 3 from sampling | Nine times the catches for 1.5 times the cost — 4.2 reviewer-hours per catch became 0.70. |
| Externally consequential errors prevented | Unknown | 18 | Fourteen figure errors, three payment-plan notices, one same-day dispute. |
The cost comparison is the number that carries a redesign through a management meeting. Quarter one consumed about 46 reviewer-hours — 18,400 items at nine seconds — and produced eleven catches, or 4.2 reviewer-hours per catch. Quarter two consumed about 69 hours: 56 on the narrow gate at three minutes an item, and 13 on the weekly sample. It produced 96 catches from the gate and three more from sampling, or roughly 0.70 reviewer-hours per catch. That is one and a half times the cost for nine times the catches, and the catches are in the class of item that reaches a customer holding a wrong figure.
Two cautions about reading those numbers, because the second quarter is easy to oversell. A rejection rate is not a target and is trivially easy to inflate — a reviewer told that 8.7 per cent looks healthy can produce 8.7 per cent by lunchtime. What makes the figure meaningful is not the rate but the structure behind it: every rejection carries a reason code, routes to a named owner, and has a recorded outcome, so the population can be examined rather than merely quoted. And the sampled defect rate of 1.5 per cent on the ungated ninety-four per cent is not a footnote. It is the entire justification for narrowing the gate, and without it the redesign would be a cost reduction wearing a governance argument.
The artefact that mattered most at the August assessment was the one nobody wanted to hand over: the quarter-one file, with its nine-second median and its eleven rejections, accompanied by a note explaining what the numbers showed and what changed on 1 May. A programme that finds its own weakness, quantifies it and fixes it reads as a functioning management system. A clean quarter with no such record reads as a programme that has not yet looked. The same asymmetry appears in how populations and samples get tested generally: a documented exception with a correction is a stronger position than a population nobody can reconcile.
One last figure from the composite, for anyone budgeting. The redesign took eleven working days end to end and roughly nine person-days of effort: two days exporting and analysing the queue, one day timing the checks with reviewers, three days rebuilding the review surface so all three checks could be performed without leaving the screen, two days building the rejection path and the reason-code list, and one day writing the control description and the sampling plan. The single largest item was the review surface, which is the pattern almost everywhere — the governance work is cheap and the interface work is not, because a checklist the reviewer cannot execute in the screen in front of them is a checklist that will be executed from memory.
What an assessor actually requests
Three parties ask, and they ask for different things. An ISO/IEC 42001 certification body at Stage 2 tests whether the management system operates, which means records rather than intentions. A licensed CPA firm performing a SOC 2 examination under AT-C 205 will only reach the gate if the AI system sits inside the system boundary described under DC 200, and there are no AI-specific trust services criteria: the gate is examined through criteria that already existed, principally the deployment of control activities through policies and procedures under CC5.3, logical access under CC6.1 to CC6.3, monitoring under CC7.2, change management under CC8.1, and — for the reviewers themselves — the commitment to competent individuals under CC1.4. An enterprise customer’s security team asks the same questions in a questionnaire, less formally and usually faster.
| The request | What satisfies it | What gets rejected |
|---|---|---|
| The complete population of actions subject to review | A system-generated export of every action meeting the gating rule during the period, with the query, filter, date range and record count on the face of it, reconciled to the system that actually performed the action. | A list of reviews. That is the population of things that were reviewed, which by construction cannot show the things that should have been and were not. |
| Evidence of gate coverage — that nothing bypassed it | The action population from the target system joined to the review population, with every unmatched row explained, plus the gate configuration captured at two or more dates inside the period. | An assertion that the workflow makes bypass impossible. A configuration screenshot taken during fieldwork, which evidences design on that date and nothing about the period. |
| For each selected item, what the reviewer was shown | The stored review record, or a versioned definition of the review surface with change dates, so it is determinable what was visible on the date of that decision. | A live link to the current interface. An approved flag with no basis attached. A surface that changed mid-period with no record of when. |
| Reviewer identity and authority | A named individual resolved from the identity provider, the delegation of authority showing that person may reject, and evidence that the approver is neither the requester nor the agent’s own identity. | A shared queue account. A role name. An approval recorded against the service principal that generated the item. |
| Decision, timing and ordering | Decision value, decision timestamp and action timestamp in a consistent zone, with the review preceding the action for every item claimed as preventive. | Approvals timestamped after the action they purport to authorise. A single timestamp. A block of decisions all bearing the same second. |
| Reviewer competence across the period | Dated training records naming the system, the version and the content, covering every individual appearing in the decision population — including anyone added mid-period. | A training deck with no attendance record. Coverage of the original rota only, tested against a population that turns out to contain five other names. |
| What happened to the rejections | Reason codes, the successor ticket or owner record, the resolution reached, and the override log where a refusal was reversed with the original refusal preserved. | Rejections with no successor record. An override that replaced the refusal. A reversal with no named approver. |
| The management review of the oversight metrics | A dated record naming attendees, the metrics reviewed, the thresholds applied, the exceptions raised, and evidence that raised items were closed. | A dashboard link. A review held after period end for a control scheduled inside it. Minutes recording attendance but no decision. |
The first two rows are the ones that separate a prepared programme from an unprepared one, and they are the same insight twice. The population that matters is the population of actions, drawn from the system that performed them — not the population of reviews, which is by construction a list of the things that reached the gate. Only the first can answer the question an assessor is really asking, which is whether anything went round it. Where the two populations cannot be joined at all, the honest position is that gate coverage is unevidenced, and that is a finding worth self-reporting rather than discovering under examination.
On the zero-rejection question, which comes up in nearly every engagement: a period containing no rejections is not automatically a deficiency, and no criterion makes a non-zero rejection rate a condition of valid evidence. But it will be inquired into, so have the explanation ready and have it corroborated from somewhere other than the queue itself — a low-variance population, a gate narrow enough that it only ever sees clean items, an upstream automated control that catches the errors before they reach a human, or a genuine quality result visible in the sampled defect rate on the ungated tier. The answer that fails is a shrug.
Population, frequency and what gets drawn
| Oversight control | How the population is defined | Typical frequency | Sample drawn | The artefact, per item |
|---|---|---|---|---|
| Pre-action review gate | Every action meeting the gating rule during the period, drawn from the system that performed the action rather than from the review queue. | Many per day | 25–60 | Per item: the review record, the named reviewer, both timestamps, and the decision. |
| Sampled assurance over ungated actions | The sample list itself, plus the ungated population it was drawn from and the selection method used. | Weekly | 5–15 weeks | The completed checklist per sampled item, including every item that produced no finding. |
| Override of a refusal | Every override in the period. Usually a small number, sometimes one. | Ad hoc | All, where small | The original refusal, the override, the stated reason, and the named approver. |
| Reviewer competence and rota | Every individual who recorded a decision in the period, taken from the decision population rather than from the rota document. | On joining and on change | All | Dated training record naming the system and the version. |
| Change to the gate configuration | Every change to the gating rule, the thresholds, the checklist or the review surface. | Ad hoc | All, where small | The change record, the approver, and the configuration before and after. |
| Management review of oversight metrics | Every scheduled occurrence falling inside the period. | Monthly or quarterly | All, or 2–4 | Minutes with metrics, thresholds, exceptions raised, and closure of raised items. |
| Test of the stop path | One. | Annual | 1 of 1 | A dated test with elapsed times and a note on what remained in flight afterwards. |
The frequencies and sample counts above are market practice, informed by the AICPA’s sampling guidance — they are not a requirement of ISO/IEC 42001 or of the trust services criteria. The certification body or the CPA firm sets its own, and may draw more where a control is new, where the population is unstable, or where an exception has already been found. Two consequences are worth planning around. The population definition is agreed before the sample is drawn, so the sentence describing it — all outbound messages where the amount-asserted flag is true or the account carries a dispute flag, for these dates, reconciled to the send log — is itself a deliverable, and a vague one costs a week. And the low-frequency rows carry no margin at all: a monthly metrics review missed once is an eight per cent exception rate in its own population, and an annual stop test missed once is a control with no evidence whatsoever.
Retention is the recurring silent failure. Review records live in whichever workflow or ticketing tool the gate was built in, and those tools retain on a plan setting rather than on a governance decision — frequently for less time than the period an assessor will ask about, and almost always for less than the period plus the lag before fieldwork begins. Where the EU AI Act applies, Article 26(6) requires deployers of high-risk systems to keep automatically generated logs under their control for at least six months unless other Union or national law provides otherwise, and the recurring mistake is reading six months as a target rather than a floor. On the day a gate goes live, list every store you rely on in the control description, write its retention window beside it, and schedule an export to storage you control wherever the window is shorter than the assessment period. It costs an afternoon then and cannot be done at all nine months later.
Eight numbers that reveal whether review is genuine
Every one of these is computable from the twelve-field record, none require new tooling, and together they answer the only question that matters about a gate: is anybody actually behind it. Put a threshold beside each and a decision beside each threshold. A metric reported without a threshold is a chart. A metric with a threshold and a recorded decision is a control, and it is the second kind that appears in management review minutes an assessor can test.
| Metric | How to define it | What a bad reading looks like | The decision it forces |
|---|---|---|---|
| Rejection rate | Decisions other than clean approval — reject and modify counted separately — as a proportion of decisions, broken down per gate and per reviewer. | At or near zero over a sustained period. Also a rate that moves the month after a throughput target is introduced. | Investigate the review surface and the checklist before concluding the population is clean. Never set the rate as a target. It is the fastest metric here to corrupt. |
| Time on task | Item-opened to decision-recorded, reported as a median and a 25th percentile, against a measured benchmark of how long the checks actually take. | A median below the time required to read the item. A 25th percentile in single-digit seconds. | Reduce what is gated. Adding reviewers moves the number without changing the incentive that produced it. |
| Gate coverage | Actions that met the gating rule and were performed with no matching review record, as a count and as a proportion of the action population. | Any non-zero figure, and any period where the join between the two populations cannot be performed at all. | Fix the enforcement point rather than the process document. A gate that can be bypassed is a guideline. |
| Review latency | Time from an item entering the queue to a decision, as a median and a 95th percentile, against the operational tolerance for that action. | A tail longer than the business can wait for. That tail is where bypass requests and standing exceptions originate. | Add capacity, narrow the gate, or grant a documented exception path — but decide, because an unmanaged tail decides for you. |
| Reviewer concentration | Share of decisions taken by the single busiest reviewer, and the count of days on which one person exceeded a stated per-day threshold. | A dominant reviewer, and days where one person cleared several times the sustainable rate. | Widen the rota. Concentration is a quality risk and a continuity risk at once, and it is invisible in every policy document ever written. |
| Override rate on refusals | Refusals subsequently reversed, as a proportion of refusals, with the approver named and the reason categorised. | A high rate, or overrides concentrated on one requester. A flat zero where the path exists usually means refusals are being avoided rather than honoured. | Read the reasons rather than the rate. A refusal path that is always overridden teaches reviewers not to use it. |
| Sample defect rate on ungated actions | Findings per sampled item within the tiers you deliberately chose not to gate, trended across periods. | A rising rate, or any externally consequential finding appearing in the sample at all. | Move the affected class into the gate. This metric is the licence for a narrow gate. Without it, narrowing is unevidenced. |
| Uncertainty and escalation usage | How often reviewers use the not-sure path, what those items turn out to be, and how long they take to resolve. | Zero usage, which almost always means the option does not exist and uncertainty is resolving as approval. | Provide the third option and staff the escalation, or accept in writing that ambiguous items are being approved. |
Read them in pairs rather than individually, because a single number is nearly always ambiguous. Rejection rate alone cannot distinguish a clean population from an absent reviewer. Rejection rate beside time on task can. Time on task alone cannot distinguish a fast expert from a fast clicker. Time on task beside the sampled defect rate can. Reviewer concentration alone looks like a staffing detail. Concentration beside review latency shows you the days when the queue was cleared by one exhausted person and the days when it sat. The pair that most reliably produces a genuine finding is gate coverage beside override rate, because between them they describe everything that got past the control — the actions that never reached it and the refusals that were undone.
One warning about targets. Every metric here can be moved without improving anything, and rejection rate can be moved in an afternoon. If any of these becomes an objective that reviewers are measured against, it will be met and it will stop being informative, which is why the throughput target is the one thing that must never be applied to a reviewer. Measure reviewers on decision quality assessed by a periodic second look, and use the eight numbers to manage the design of the gate rather than the behaviour of the people standing at it.
The nine cases that generate the real arguments
The design above handles the common cases cleanly. These nine produce genuine disagreement between security, engineering, legal and the business, and settling them in advance is worth several days later.
The system is right more often than the reviewer
This is the uncomfortable case and it is not rare. Where a model outperforms the average reviewer on the same task, inserting a gate can lower aggregate accuracy — the reviewer overrides correct outputs more often than they catch wrong ones. The honest response is to separate two arguments that usually get merged. Oversight for accuracy has to be justified by measurement: run the reviewer and the system against a labelled set and see. Oversight for accountability and recourse does not depend on accuracy at all. It exists so that a person is answerable for the outcome and an affected individual has someone to appeal to. If your gate is doing the second job, say so plainly and stop defending it on the first, because the accuracy argument is the one that loses.
Volume arrives faster than the rota can grow
Seasonal peaks, a marketing campaign, a competitor outage — the gated population triples and the rota does not. Decide the degradation mode in advance and write it into the control description, because the alternative is that it gets decided at 4 p.m. by whoever is under most pressure. Two defensible options exist: narrow the gate temporarily by raising the threshold, as a recorded and time-boxed change with the accountable owner as approver, or throttle the system so it produces less. The option that is not defensible is keeping the gate nominally in place while approving at four times the speed, because that produces a record showing the control operated throughout, which is worse than a record showing it was suspended.
The vendor’s product is the gate
Where the review step lives inside a third-party product, the record you need may not be yours. Five questions settle it: is the review record exportable, how long is it retained, does it capture what the reviewer was shown rather than only the decision, does it resolve the reviewer to a named individual, and can you obtain the history after termination. Treat the answers as contract terms and confirm them in current vendor documentation rather than in a sales deck, because retention and export behaviour differ between plan tiers and change between releases. Where the answers are thin, compensate on your own side: whatever the approved action does inside your systems is recorded by your systems, and that is the population an assessor can actually test.
The agent runs at 2 a.m. and nobody is there
Unattended operation is a design choice, not an accident, and the gate cannot simply be asserted into it. Three honest configurations exist. Queue the consequential actions for release the next morning, accepting the delay. Bound the run with ceilings, velocity limits and circuit breakers so that no single night can exceed a stated blast radius. Or wake someone, which is a real option for a genuinely small number of action types. Choose explicitly and describe it accurately. The failure is a control description claiming human review of a process that completes at 02:06 and is first seen at 09:20 — a gap that is defensible when described and indefensible when discovered.
The reviewer benefits from approving
A sales representative approving AI-drafted discounting, an engineer approving their own agent’s pull request, a manager approving a recommendation that hits their own target — the conflict is structural rather than personal, and it will not be fixed by asking people to be careful. Enforce the separation in configuration, not in policy: the approver may not be the requester, may not be the beneficiary of the metric the action moves, and may not be the identity that generated the item. Where the organisation is genuinely too small for separation, say so, and compensate with sampled review by someone outside the process on a defined rate. A stated compensating control reads far better than an undisclosed conflict.
One AI system reviews another
Using a model as a checker is useful and increasingly common, and it is not the oversight control. It shares failure modes with the system it checks, it can be induced to approve by the same content that induced the error, and it cannot bear accountability, which is half of what oversight is for. Describe it accurately as an automated detective control that filters the queue — a good one, worth building, and evidenced like any other automated control through its configuration and its output. Then keep the human gate on the consequential tier. The value of the model checker is that it makes the human tier small enough to be real.
The decision is legally significant
Where an output produces legal effects or similarly significantly affects a person, the analysis moves onto different ground. Under the GDPR, guidance from the Article 29 Working Party is explicit that oversight must be meaningful rather than a token gesture, and must be carried out by someone with the authority and competence to change the decision — with human employees rubber-stamping automated profiles specifically named as insufficient. The Court of Justice held in the SCHUFA case (C-634/21, 7 December 2023) that a score can itself constitute an automated decision where the recipient draws strongly on it. The practical consequence: the record must show not only that a person decided, but that they could have decided otherwise and had what they needed to.
You inherited a hundred per cent gate you cannot narrow
Sometimes a customer contract or a regulator requires review of every output, and the narrowing argument is unavailable. The design problem then moves inside the gate: tier the depth rather than the coverage, so that every item receives a recorded decision but only the consequential tier receives the full checklist, with the remainder receiving a fast structured check whose scope is documented. Add sampling as a second layer over the fast tier. And write a throughput plan with real numbers, because the clause you are honouring was almost certainly drafted without one. At renewal, the queue export and the tiered defect rates are the strongest argument available for changing the clause.
The gate was there and the incident happened anyway
An approved item causes harm. What the record does next depends entirely on what is in it. A record showing three minutes on task, the checks performed, the reviewer named and trained, and a judgement that turned out wrong describes a control that operated and a human who erred — recoverable, explicable, and the reason the control was designed with a detection layer behind it. A record showing nine seconds describes a control that did not operate, and the incident becomes a governance failure rather than a mistake. This asymmetry is the strongest practical argument for capturing time on task, and it is one people only appreciate afterwards.
Two of these have close relatives elsewhere in this wing. Where untrusted content can reach a system that acts, the gate has to be designed against prompt injection once tools are connected — a reviewer approving a summary that was itself written by injected content is the purest form of reviewing the machine’s account of itself. And where the deployment is unattended, the boundary question of when a tool becomes an agent decides whether a gate is available at all.
What people object to and the answer
Eight arguments come up in nearly every programme. In every case the difference between a good answer and a defensive one is a number.
“A human reviews everything before it goes out, so we are covered.”
Reviewing everything is the weakest form of this control, not the strongest, and it is worth explaining why before someone external does. Universal review distributes a fixed amount of attention across the whole population, which means the items that genuinely needed scrutiny receive exactly the same treatment as the ones that did not. The claim is also testable in about twenty minutes: export the queue, compute the median time on task, and compare it to the time it takes to read one item. If the first number is smaller, the control being described does not exist. Narrowing the gate is what makes the remaining reviews real.
“Our approval rate is a hundred per cent, which shows the model is accurate.”
It shows one of three things and the record does not distinguish between them: the population really is clean, the gate is too narrow to see anything interesting, or nobody is looking. Those have very different consequences and they are separated by other evidence — the sampled defect rate on ungated items, the reviewer’s time on task, and whether a rejection path exists and is usable at all. Bring that evidence yourself. An organisation that says “our rate is very low and here is how we satisfied ourselves it is genuine” is in a much stronger position than one that offers the rate as though it settled the question.
“The queue is backed up. We will add another reviewer.”
That buys throughput at linear cost and leaves the structure untouched. If the gated population grows — and with AI-generated output it grows quickly — you are back in the same position in a quarter, having doubled the run rate. Work the arithmetic instead: how many items meet the gating rule, how long the checks genuinely take when timed, and what capacity you actually have at realistic occupancy. Usually the gated population contains a large majority of items that are reversible and internal, and moving them to sampled assurance restores the gate without adding a person. Add reviewers when the consequential population is genuinely large, not to absorb a scoping failure.
“Show us that a human approves high-risk actions.” (a buyer)
This is a reasonable and testable request, so answer it with the population rather than the policy: the count of actions that met the gating rule, the reviews performed, the median time on task, and the proportion modified or rejected. It is worth saying out loud that a ninety-two per cent approval rate is a stronger answer than a hundred per cent one, because a reviewer who never changes anything is either unnecessary or not reading. A security team that has run these reviews internally will recognise the honesty immediately, and it converts a compliance question into a technical conversation between people who understand the same problem.
“Give the reviewer the model’s confidence score so they know when to look harder.”
Reasonable in intent and frequently counterproductive in effect. A confidence score becomes a triage instruction, and reviewers rationally spend their attention where the score is low — which concentrates scrutiny exactly where the system already knows it is uncertain and withdraws it from confident, wrong outputs, the class that causes most harm. If the score is shown, calibrate it and tell reviewers what it measures and what it does not, and monitor whether rejections cluster on low-confidence items. Where the decision is consequential, consider capturing the reviewer’s own reading before revealing the score at all.
“The person who requested the output knows it best, so they should approve it.”
They do know it best, and that is not the property the control depends on. Oversight exists to introduce a perspective that is not already committed to the output, and the requester is committed by construction — they asked for it, they need it, and rejecting it costs them time they have already spent. This is the segregation of duties argument that predates AI entirely, and the automated version is worse because the volume is higher and the commitment is cheaper. Enforce non-self-approval in configuration and record the prohibition, because it is among the first things a buyer’s security team probes.
“This will slow us down.”
Sometimes, and the number is knowable rather than rhetorical, which changes the conversation. Measure the two figures that matter: how much of the population is actually gated, and what the median and 95th-percentile latency are on that subset. A well-scoped gate typically touches single-digit percentages of traffic and adds under an hour to those items. If the gate touches everything and adds a day, the objection is correct and the design is wrong. The teams that handle this best publish the tiering rule in advance, so a team can predict which path its work will take before it builds — which is the difference between governance that shapes a design and governance that arrives to block one.
“We are not in the EU, so Article 14 does not apply to us.”
Possibly true as a matter of jurisdiction and largely irrelevant as a matter of design. The EU AI Act applies by placing on the EU market or by output being used in the Union, so the territorial question deserves an actual answer rather than an assumption. But the five oversight capabilities in Article 14(4) were drafted against precisely the failure modes on this page, which is why they are the best short checklist available regardless of whether the Act binds you. Buyers increasingly ask against them by name. And ISO/IEC 42001, which is certifiable anywhere, requires responsible-use processes and defined competence that land in the same place through different words.
The same gate, in three vocabularies
The three regimes do different jobs and conflating them is the most common error in an AI governance paper. The EU AI Act is regulation, and where it applies it is binding and enforced by authorities. ISO/IEC 42001:2023 is a management system standard, certifiable by an accredited certification body, and it requires you to have decided rather than telling you what to decide. SOC 2 is an attestation performed by a licensed CPA firm under AT-C 205, and it contains no AI-specific criteria at all — an AI gate inside the system boundary is examined through criteria that already existed. The table is the mapping used to answer the same question in whichever vocabulary the asker brought.
| The design question | EU AI Act, Regulation (EU) 2024/1689 | ISO/IEC 42001:2023 | SOC 2, where the system is in scope |
|---|---|---|---|
| Which actions require review | Article 14(3): oversight measures commensurate with the risks, the level of autonomy and the context of use. | Processes for responsible use (A.9.2), objectives for responsible use (A.9.3) and intended use (A.9.4). | CC5.1 and CC5.3 — control activities selected against assessed risk, and deployed through policies and the procedures that put them into action. |
| What the reviewer must check | Article 14(4)(a) and (c): understand capacities and limitations, and correctly interpret the output. | AI system operation and monitoring (A.6.2.6). | CC2.1 — relevant, quality information obtained or generated and used to support the functioning of internal control. |
| Reviewer competence and authority | Article 14(4)(b) and (d), and Article 26(2), which requires deployers to assign oversight to natural persons with the necessary competence, training and authority, as well as the necessary support. | Competence under clause 7.2 and awareness under 7.3. Roles, responsibilities and authorities under clause 5.3 and A.3.2. | CC1.4 on attracting, developing and retaining competent individuals. CC1.3 on structures, reporting lines and appropriate authorities. |
| Refusal, intervention and stopping | Article 14(4)(d) and (e): disregard, override or reverse the output, and interrupt or halt the system in a safe state. | Operational planning and control under clause 8.1, and corrective action under clause 10.2. | CC8.1 where the gated action is a change, and the incident-response criteria where a refusal escalates into one. |
| Evidence that oversight operated | Article 26(6): deployers of high-risk systems keep automatically generated logs under their control for at least six months, unless other Union or national law provides otherwise. | Documented information under clause 7.5, and AI system recording of event logs (A.6.2.8). | CC4.1 evaluations, plus the standing expectation that information produced by the entity is complete, accurate and testable. |
| Revisiting the design when things change | Article 14(3) again, because proportionality is assessed against the deployment as it actually is rather than as it was approved. | Monitoring under clause 9.1, management review under 9.3, corrective action under 10.2, and reassessment on significant change. | CC8.1 change management, with CC4.1 for the ongoing evaluation that catches drift between them. |
Two cautions about reading it. The SOC 2 column applies only where the AI system sits inside the boundary described under DC 200. Where it does not, the honest answer to a buyer is that the report does not cover it, which is a better answer than a mapping that implies otherwise. And the ISO column names controls and clauses rather than prescribing a gate, because the standard genuinely does not prescribe one — anybody quoting a clause number as though it mandated a specific review design is reading something into it that is not there. The controls themselves are set out in the Annex A control reference.
Six things to do in the order that pays
In roughly this sequence, because each step makes the next one cheaper. The first two cost a day between them and change the argument entirely, since they replace opinions about whether the gate is working with two numbers that settle it.
- 01 Export one month of your review queue and compute the median and 25th-percentile time on task. Then time how long it takes to read one item. If the first number is smaller than the second, you have found the finding before anyone else did, and everything below follows.
- 02 Time the checks. Sit with two reviewers and have them perform the verification they believe they are doing, against the real source systems, with a stopwatch. That number is your true per-item cost and it is almost always several times what the design assumed.
- 03 Write the gating rule as a query over fields the system already holds. If the rule cannot be expressed as a query, it cannot be enforced in configuration, and a rule that depends on someone remembering is not a gate.
- 04 Replace “review for accuracy” with three named checks, each verifiable against something the model did not write, and rebuild the review surface so all three can be performed without leaving the screen.
- 05 Build the rejection path before you rebuild the gate: destination, owner, no-resubmission-without-change, reason codes, and an override that preserves the original refusal. A gate with no working exit is a very expensive approve button.
- 06 Put the eight metrics in front of the accountable owner on a stated cadence with a threshold beside each. A metric without a threshold is a chart. A metric with a threshold and a recorded decision is a control.
Where this becomes part of a certifiable management system, the machinery is ISO/IEC 42001:2023: a defined scope, an AI policy, allocated roles and authorities, AI risk and impact assessments, responsible-use processes with stated objectives and a documented intended use, monitoring under clause 9.1, internal audit under 9.2, management review under 9.3 and corrective action under 10.2. The gate design on this page is what makes several of those requirements answerable about a specific queue on a specific Tuesday, which is the part teams find hard. The wider framing sits in the ten questions that govern any AI agent, and the scale problem — one gate is a design, four hundred gates is an operating model — in governing many agents at once.
A boundary worth stating plainly. Tranquility Cybersecurity advises, implements and coordinates: we design the gate, write the control description, build the evidence model and run the readiness work across 20+ frameworks for clients in the USA, UK, Australia, UAE and India. ISO 42001 certificates are issued by accredited certification bodies. SOC reports are issued by licensed CPA firms.
Frequently Asked Questions
What actually makes human oversight a control rather than a workflow step?
What is a good rejection rate?
Is a hundred per cent approval rate a finding?
Does the EU AI Act apply to us if we are not in the EU?
How long should a review take?
Can another AI system perform the review?
What evidence proves a human actually reviewed something?
Should the reviewer see the model’s confidence score or explanation?
What if the model is more accurate than our reviewers?
Does ISO 42001 require human-in-the-loop?
Related Reading
Copilot Oversharing and Permission Hygiene
AI collapses the effort to discover information, so latent over-permissioning becomes exploitable.
Read moreConnectors and the Permission Inheritance Problem
Data access entitlement is not authorisation for an AI purpose. Assessing connectors proportionately.
Read moreHow Much Should an AI Agent Be Allowed to Spend?
Per-transaction ceilings are the easy part. Velocity and aggregate limits are what actually bind.
Read moreThe Ten Questions That Govern Any AI Agent
Scope, ownership, data, capability, identity, oversight, failure detection, evidence, reassessment and how to stop it.
Read moreISO 42001 Knowledge Hub
AIMS controls, EU AI Act mapping, risk assessment and guides.
Read moreISO 42001 (AI Management)
The world's first AI management system standard, for AI builders.
Read more