Levels of Autonomy and Human-in-the-Loop Design for AI Agents: Matching Oversight to Risk and Earning Autonomy with Evidence

1. Introduction

Until recently, the question of how much an AI system should do on its own had a simple answer for most businesses: very little. A language model drafted text and a person decided what to do with it. AI agents change that. An agent plans a sequence of steps, calls tools, reads the results and continues until the task is done, and it can post an invoice, send an email or change a record without anyone pressing a button. The capability behind this is growing quickly. Researchers at METR measured the length of software and research tasks, in human working time, that frontier models complete with 50 percent reliability, and found that this time horizon has been doubling roughly every seven months since 2019 [1].

Capability, however, does not decide autonomy. Feng, McDonald and Zhang argue that the level of autonomy at which an agent operates is a deliberate design choice, separate from what the agent is able to do and from the environment it works in [2]. A highly capable agent can be deployed as an assistant that proposes and waits, and a modest one can be allowed to act alone on narrow, low-risk tasks. The question for a business is not "how good is the model?" but "which actions may this agent take without a person, and how do we know?"

This paper addresses that question. Section 2 reviews the levels-of-automation literature and maps it to agents. Section 3 summarizes the evidence on why human oversight often fails in practice. Section 4 proposes setting autonomy per action using four properties of the action, with a simple cost rule for when an approval step is worth its cost. Section 5 describes how to design the human checkpoint so that oversight is real. Section 6 sets out an evidence-based path for promoting an agent from one level to the next, and for demoting it.

2. Levels of automation: from teleoperators to agents

2.1 Classic scales

The idea that automation comes in degrees is old. Sheridan and Verplank, studying the remote control of undersea vehicles, described ten levels running from a computer that offers no assistance, through one that suggests alternatives or executes a choice if the human approves, to one that acts entirely on its own and informs the human only if it decides to [3]. Their scale introduced the distinctions that still matter for agents: who generates the options, who selects one, who executes it, and whether the human is told.

Parasuraman, Sheridan and Wickens refined this into a two-dimensional model [4]. They separated four functions that a system can automate (information acquisition, information analysis, decision selection and action implementation) and argued that each can be automated to a different level. A system can gather and analyze information with high autonomy while leaving the decision to a person. Their central recommendation was that the right level for each function should be set by the consequences of errors and by human performance, not by what the technology makes possible.

The best-known scale today is SAE J3016 for driving automation, with six levels from no automation (Level 0) to full automation in all conditions (Level 5) [5]. Its key contribution is the notion of an operational design domain: a Level 4 vehicle drives itself without a human fallback, but only within defined conditions such as a mapped area or certain weather. Outside them the system must hand over or reach a safe state.

2.2 A scale for AI agents

Feng and colleagues propose five levels for AI agents defined by the role the user plays [2]. At Level 1 the user is an operator who directs the work and makes the decisions while the agent acts on command. At Level 2 the user is a collaborator who shares planning and execution with the agent. At Level 3 the user is a consultant whom the agent asks for input and preferences. At Level 4 the user is an approver whose consent is needed for certain actions, such as high-risk ones. At Level 5 the user is an observer who can monitor the agent and stop it but does not intervene in normal operation. Table 1 aligns these scales.

The analogy with driving should not be pushed too far. A vehicle performs one task, driving, so a single level describes the system. An agent in a business performs many different actions with very different consequences: reading a record, drafting a reply, issuing a refund, deleting a file. A single autonomy level for the agent as a whole is therefore too coarse. The Parasuraman model, which assigns levels per function, is closer to what agents need, and Section 4 extends it to per-action levels.

3. Why human oversight fails

The default response to agent risk is to put a human in the loop. Decades of research on automation show that this is necessary but not sufficient: a human in the loop provides real protection only under specific conditions.

3.1 The ironies of automation

Bainbridge observed that automation creates two ironies [6]. First, designers automate the tasks they can and leave the human to handle what they cannot, so the human is left with the hardest and rarest situations. Second, the human's role becomes monitoring, a task people perform poorly over long periods, and the skills needed to take over decay precisely because the automation usually works. Both apply to agents. If an agent handles routine invoices and escalates only the unusual ones, the clerk who reviews escalations sees only difficult cases while losing the daily practice that built their judgment.

3.2 Out of the loop

Endsley and Kiris showed experimentally that operators who monitored full automation were slower and less accurate when they had to take over than operators who had been working at an intermediate level of automation [7]. The cause was reduced situation awareness: the monitoring operator had not been building a model of the situation. For agents, the lesson is that a person who is asked to approve a single action at the end of a long chain, with no view of how the agent got there, cannot judge it well.

3.3 Automation bias and complacency

Parasuraman and Riley distinguished use, misuse, disuse and abuse of automation, where misuse means relying on automation when it should not be relied on [8]. A systematic review of automation bias in clinical decision support found that users frequently accepted incorrect advice, that errors of commission (following wrong advice) and omission (missing a problem the system did not flag) both occurred, and that the effect was stronger when workload was high and when users had greater trust in the system [9]. Lee and See framed the design goal as appropriate reliance: trust should match the automation's actual reliability in the situation at hand, which requires that the system make its reliability visible [10].

3.4 Approval fatigue

These findings combine in a pattern familiar to anyone who has deployed an approval workflow. When an agent is usually right, approvals become a formality. The reviewer sees a long queue of similar requests, the first hundred are correct, and the hundred-and-first is approved without being read. The approval step then adds delay and cost while providing little protection, and it creates a record that suggests oversight took place.

The EU Artificial Intelligence Act reflects this research. For high-risk systems, Article 14 requires that the people assigned to oversight be able to understand the system's capacities and limitations, remain aware of the tendency to rely too much on its output, which the Act calls automation bias, correctly interpret the output, decide not to use it or override it, and interrupt the system through a stop button or similar procedure [11]. These are useful design requirements even where the Act does not apply.

4. Setting autonomy per action

4.1 Four properties of an action

The appropriate level of autonomy depends on the action, not the agent. This paper proposes four properties for classifying each action an agent can take.

1. Reversibility. Can the action be undone fully and cheaply? A draft can be deleted; a payment to a third party, an email to a customer or a deleted record without a backup cannot.
2. Impact. What is the worst plausible consequence, in money, safety, legal exposure or reputation? Impact can be bounded by limits, for example a maximum refund value.
3. Detectability. If the action is wrong, how likely is the error to be caught before it causes harm, by a downstream check, a reconciliation or the recipient?
4. Novelty. Is the situation within the range of cases on which the agent has been tested, or outside it? This is the agent equivalent of SAE's operational design domain [5].

Table 2 applies these properties to common agent actions and gives a default level.

The table encodes a few principles. Read-only actions can run with high autonomy because their only risk is a wrong answer, which is controlled by evaluation and by the reader. Actions with external effects need more oversight than internal ones. Impact can be capped by limits, which turns a high-impact action class into a bounded one. And some actions, notably those affecting physical safety, should not be delegated at all; an agent can advise on them through the systems people already use.

4.2 When an approval step pays for itself

An approval step has a cost: the reviewer's time and the delay. It also has a benefit: the errors it catches. A simple expected-loss comparison makes the trade-off explicit. Let p be the probability that the agent's action is wrong, C the cost of an error that reaches the world, d_a the probability that an error is caught without a reviewer (for example, by a downstream reconciliation), d_h the probability that it is caught with a reviewer, and c_r the cost of one review. Approval is worthwhile when the loss it avoids exceeds its cost:

Consider refunds. Suppose the agent is wrong on 2 percent of refunds, an uncaught wrong refund costs $300 on average, a monthly reconciliation would catch 30 percent of errors anyway, and an attentive reviewer catches 90 percent. The left side of Eq. (1) is 0.02 × 300 × 0.6 = $3.60 per refund. If a review takes two minutes of a clerk's time at $40 per hour, the cost is about $1.33, so approval pays. If the error rate falls to 0.5 percent after improvements, the benefit drops to $0.90 and approval no longer pays for small refunds; a sample review is better. The same arithmetic shows why limits matter: for a refund above $2,000 the benefit of approval is large even at a low error rate.

4.3 Can the reviewers keep up?

Eq. (1) assumes the reviewer's detection rate d_h. The research in Section 3 suggests that this rate falls when reviewers are overloaded or when nearly every request is correct [9]. A second check is therefore a capacity constraint. If an agent generates N approval requests per day, each needing t minutes of genuine attention, and the reviewers can give T minutes a day to approvals, then:

When Eq. (2) is violated, reviews will be rushed and d_h in Eq. (1) will fall toward d_a, eliminating the benefit. The remedies are to reduce N by raising autonomy for low-risk actions, to reduce t by presenting better evidence, or to replace blanket approval with sampling. Figure 1 shows the resulting regions.

5. Designing the human checkpoint

If approval is required, it should be designed so that the reviewer can actually exercise judgment. Five design elements follow from the research in Section 3.

5.1 Show the evidence, not just the action

The reviewer should see what the agent proposes to do, the inputs it used, the rule or precedent it relied on, and anything unusual about the case. For a change to a record, this means a before-and-after comparison. For a payment, it means the invoice, the purchase order and the receipt side by side, with discrepancies highlighted. This addresses the situation awareness problem identified by Endsley and Kiris [7] and lets the reviewer calibrate trust to the case at hand rather than to the agent in general [10].

5.2 Ask for a decision, not a click

Interfaces that default to approval encourage automation bias. Requiring the reviewer to confirm the specific value that matters (the amount, the recipient) or to choose between approve, edit and reject with no default makes it harder to approve without reading. Recording the reason for each rejection gives the data needed to improve the agent.

5.3 Sample what is not approved

Actions that run without approval should be sampled. A daily random sample of autonomous actions, reviewed with the same care as an approval, measures the agent's real error rate and keeps reviewers familiar with normal cases, which counters Bainbridge's second irony [6]. Sampling can be weighted toward higher-value or more novel cases.

5.4 Escalate on signals, not only on categories

The agent should escalate to a person when it meets conditions it was not tested on: a missing document, a value outside the historical range, conflicting instructions, a counterparty it has not seen, or low confidence in its own output. Anthropic's guidance on building agents recommends this kind of explicit stopping condition and human checkpoint, together with starting from the simplest design that works [12].

5.5 Make stopping easy and logging complete

Every agent should have a stop control that takes effect immediately, a way to roll back recent actions where they are reversible, and an audit log that records each action, its inputs, the identity under which it ran and who approved it. These are the operational counterpart of the oversight requirements in Article 14 of the EU AI Act [11] and of the MANAGE function in the NIST AI Risk Management Framework, which calls for mechanisms to supersede, disengage or deactivate AI systems that perform outside their intended use [13]. The identity and logging design is covered in detail in a companion paper [14].

6. Earning autonomy with evidence

6.1 Reliability is not a single number

An agent that completes a task correctly once may not complete it correctly every time. Yao and colleagues introduced the pass^k metric, the probability that an agent succeeds on all of k independent attempts at the same task, and found that even strong agents were inconsistent: in a simulated retail customer-service domain, a leading model succeeded on all eight attempts in fewer than 25 percent of tasks [15]. For autonomy decisions, consistency matters more than peak performance, because an autonomous agent repeats the same kind of action thousands of times.

6.2 How many clean trials are enough?

Promotion to a higher level of autonomy should rest on observed performance in conditions close to production. The most practical method is a shadow-mode trial, in which the agent prepares each action, a person does the work as usual, and the two are compared. A useful statistical guide is the rule of three [16]: if n independent trials produce no errors, the upper bound of the 95 percent confidence interval on the true error rate is approximately

For example, 300 shadow-mode invoices with no material error support a claim that the error rate is below about 1 percent with 95 percent confidence. Thirty clean trials support only a claim below 10 percent. Eq. (3) gives a minimum trial size for any target error rate and makes plain why a short pilot cannot justify full autonomy. Figure 2 plots the bound.

Two cautions apply. Eq. (3) assumes the trials are independent and representative; a month of trials in a quiet season may not represent year-end. And errors are not equally serious, so the trial should count material errors against the target and track minor ones separately.

6.3 A promotion and demotion path

Combining these elements gives a promotion path for each action class (Figure 3). An action starts at the level in Table 2 or one level lower. It is promoted when three conditions hold: the number of trials meets the bound from Eq. (3) for the target error rate, the errors that did occur were minor and have been fixed, and the business owner of the process signs off. Promotion is one level at a time.

Demotion should be automatic. An agent returns to the previous level when a material error occurs, when its inputs move outside the range it was tested on, or when anything in its configuration changes: a new model version, a changed tool, revised instructions or a change to the business process. Each of these invalidates the evidence that justified the current level, much as a change outside its operational design domain requires a driving system to hand back control [5]. Treating changes as demotion triggers also gives a natural place for regression testing.

Table 3 illustrates the path for an accounts payable agent. The record it produces is also the evidence an auditor, a customer or a regulator will want to see: what the agent was allowed to do, on what basis, and how that changed over time.

7. Conclusion

Autonomy is a design decision, and it should be made per action rather than per agent. The levels-of-automation literature offers a vocabulary for that decision, and the research on human oversight explains why the default answer, a human in the loop, so often fails: reviewers lose situation awareness, defer to the system and fall behind the queue.

The approach proposed here has four parts. Classify each action by reversibility, impact, detectability and novelty, and set a default level. Require approval only where Eq. (1) shows it pays for itself and Eq. (2) shows reviewers can keep up, and use sampling and monitoring elsewhere. Design the checkpoint so the reviewer sees evidence and makes a real decision. And let the agent earn autonomy through trials sized by the rule of three, one level at a time, with automatic demotion when the evidence no longer holds. Together these make oversight proportionate, keep people in the loops where they add value, and produce a record that shows why an agent is trusted to do what it does.