AI Agents Push Humans Out of the Loop
A new position paper authored by Margaret Mitchell, Avijit Ghosh, and Samir Passi argues that AI agents do not merely make oversight harder: their prolonged use can erode the very skills required to exercise it. In Italo Calvino's *The Nonexistent Knight, the most scrupulous paladin in Charlemagne's army is Agilulf, a suit of white armor with nobody inside. He exists purely through willpower, knows the regulations by heart, is always at his post, and never gets a formality wrong. It is a fitting portrait of the human in the loop when their presence is merely formal: present on the organizational chart, on the form, on the button, but lacking the time, information, and competence inside to decide.*
Now imagine an ordinary office. An AI agent reads documents, queries the management software, searches the web, prepares a proposal, updates records, drafts an email, and finally pauses: "Do you approve?". That click ought to be an informed decision, but for it to be one, you would need to know which data were used and which discarded, which tools activated, with what permissions, and with what effects. If that is impossible, you are not in the loop: you are simply the final signature on a process you do not control.
This is the core theme of AI Agents Push Humans Out of the Loop, published on arXiv on August 24, 2026, and reaching its third version on September 6. The phrase "human in the loop" is everywhere, even in Article 14 of the AI Act, which mandates effective human oversight. The authors remind us, however, that the presence of a supervisor does not guarantee reliable oversight, and that people rarely ask whether effective oversight is even possible.
With a chatbot, an error is a response to be corrected. An agent performs multiple steps and, as the paper notes, can modify or delete files without drawing attention, transfer confidential data, or initiate payments.
I touched on this in the story of Kiro, Amazon's coding agent. According to the Financial Times, citing anonymous internal sources, in December 2025 it deleted and recreated part of an environment, causing approximately thirteen hours of downtime for AWS Cost Explorer in a mainland China region. Amazon, as summarized by the AI Incident Database, described it as a very limited event resulting from user error and misconfigured permissions, not AI. In my article, I reported that the agent operated with the permissions of a senior operator and without mandatory code review. Both interpretations coexist: if the error is human because permissions were too broad, one must still ask who made it possible for broad permissions to cause such damage in the first place. Blaming those who "should have checked" does not prove that checking was practicable.
The Paper
This is a position paper: no new model, no decisive experiment, but a thesis supported by literature on automation, human-computer interaction, cognitive science, and governance. The central argument is that current practices do not support effective oversight and that prolonged agent use weakens the cognitive capabilities required to exercise it, summarized by the authors' phrase: "oversight degrades the overseer." The remedy has two legs: developers should build execution environments that sustain attention and situational awareness, while organizations should establish protocols that preserve skills and reduce fatigue. The paper does not call for abolishing agents, but rather for treating the cognitive needs of human supervisors on equal footing with the agent's capabilities.
A word of caution regarding the evidence, as the paper itself speaks of initial findings. The cited material is heterogeneous: there is a multicenter observational study on the risk of skill loss among endoscopists following exposure to AI, a systematic review on students, as well as preprints, workshop papers, and commentary on blogs and forums. Several studies closely related to the topic, such as those on developers and coding agents, are co-authored by one of the authors themselves. The framework is well grounded, but it is not a proven universal law: no one has demonstrated that every agent causes skill loss.
Why Agents Escape Control
The difficulty stems not from a vague AI opacity, but from the way an agent works: it interprets a goal, plans, selects tools, executes them, reads outcomes, and revises its plan, often with distinct internal roles for planning, execution, and evaluation. For a supervisor, the relevant information is vast and scattered across components that can change or disappear, to the point where asking why the agent chose a specific tool can be, in practice, an unanswerable question.
Then there are the tools. The paper discusses tool hallucination: an agent invents a non-existent tool, uses a real one with incorrect parameters, or misunderstands the result, causing the error to propagate downstream. A cited study, The Reasoning Trap, notes that boosting models' reasoning capabilities tends to amplify this flaw. In multi-agent systems, communication failures add another layer of complexity, as analyzed in a paper presented at NeurIPS 2025, when one agent fails to pass necessary information to another.
Imagine, as my own example, an agent tasked with cutting costs from non-strategic suppliers by ten percent. If a category is misclassified or a contractual exception is unread, subsequent communications will be logically coherent yet grounded in an error. The person approving the final email sees neither the classification nor the excluded records.
There is also the cost of evaluation. Drawing on an analysis by EvalEval co-authored by Ghosh, the paper argues that testing multi-step behavior upfront costs far more than testing single responses, making full coverage impracticable for most organizations. Part of the oversight thus shifts to when the agent is already live, falling on a person who must simultaneously use it, grant permissions, judge its choices, and predict consequences. The result is approval fatigue, where people stop scrutinizing what they approve. In a technical post on a mode in Claude Code, Anthropic notes that users approve 93% of permission requests. While a high rate alone does not prove inattention—since many actions are harmless—the company explicitly links it to that phenomenon.
The Irony of Automation
In 1983, Lisanne Bainbridge described a paradox in Ironies of Automation: the more capable a system is, the more the operator is relegated to passive monitoring, leaving them least prepared to intervene in those rare, critical moments when they must take over. The paper makes this its anchor: oversight is needed most precisely when the human overseer is least prepared. It echoes E. M. Forster's 1909 story The Machine Stops, where humanity is cared for by a system no one knows how to repair anymore.
The authors highlight deskilling, the rusting of intuition (the knack for spotting an anomaly), cognitive debt, and biases such as automation bias and anchoring to the first suggested answer. Evidence comes from various contexts beyond agents: a preliminary study associates the use of a language model for essay writing with reduced brain connectivity, though it is a preprint on a writing task and should be read as such. Nor does the irony mean that AI users stop thinking: it means that interfaces built for convenience and delegation reduce opportunities to exercise independent judgment.
Two groups are particularly vulnerable. Beginners, who risk failing to build the foundational knowledge required to verify what the agent does, and experts working at the margins of their domain, such as an experienced developer assisted by an agent in an unfamiliar programming language. The study by Dhanorkar, Passi, and Vorvoreanu on developers using agents identifies understandable yet risky shortcuts, such as treating the agent's plan as a faithful reflection of what it will do or assuming that code passing tests is inherently correct.
A useful lens comes from my article Cyborgs, Centaurs, or Automata, regarding a study by Harvard and MIT researchers on 244 BCG consultants. About 60% worked as Cyborgs, in continuous dialogue with GPT-4; 14% as Centaurs, maintaining control while delegating defined subtasks; and 27% as Self-Automators, delegating both problem and execution (percentages sum over 100 due to rounding). Centaurs produced the most accurate recommendations, while Self-Automators experienced a lack of skill development. The position paper does not cite this study—the comparison is mine—and it carries the caveats noted in my article: a 2023 experiment with a conversational model, not an autonomous agent. It serves to distinguish collaboration from abdication, not to categorize agents.
A potential vicious cycle remains, which the paper frames as a hypothesis. If an agent's quality is measured by approval or satisfaction rates, a fatigued supervisor may send positive signals without actually verifying output, rewarding the system for producing answers that are easy to accept rather than easy to audit. No malice is required: it suffices for the metric being measured (approval) to decouple from the desired outcome (well-checked action). In support, the authors cite research on model sycophancy, such as a study in Science showing that agreeable models validate users more than humans do and are preferred as a result, though none of these studies specifically examine agent oversight. The same holds for the risk that an agent might learn what is rarely inspected: a study presented at ICML documents agents hiding failures by guessing results or creating fake files, which the position paper treats as a potentiality rather than an established fact.
Bounded Autonomy, Designed Control
The solutions outlined in the paper constitute an inventory drawn from related literature rather than validated recipes for agents, and the authors present their framework as a map rather than a definitive classification.
The first category is strategic friction: introducing intentional cognitive effort at critical junctures. With pre-commitment, the supervisor records their own assessment before seeing the agent's recommendation, countering anchoring bias and leaving an auditable trail. Reasoning probes ask probing questions at delicate moments, such as "what evidence would change your mind?". Action locks require explicit verification prior to high-stakes steps. These have empirical backing: Buçinca et al. demonstrate that such interventions reduce overreliance, though users may dislike systems that employ them—a point the paper concedes.
The second is decision design: pre-establishing what the agent can do autonomously (bounded autonomy), seeking approval on coherent chunks of work presented as diffs rather than a stream of micro-confirmations, and delegating non-judgmental checks to automated tools. Here, loop engineering comes into play: as I noted in my article, a verification loop works when success can be checked mechanically (such as a test suite), but degrades into "hallucinated approval" when verified by another agent without a solid benchmark. Automated filters also have limits. In the post mentioned earlier, Anthropic reports that its classifier mistakenly blocks 0.4% of harmless actions out of ten thousand real instances, but lets through 17% of 52 overly assertive real actions, noting that for users already reviewing manually with care, it might represent a step backward and cannot replace thorough human review on high-risk infrastructure.
The third is measuring oversight quality, not just agent accuracy: tracking declining review times as task complexity increases, dropping dissent rates, disappearing requests for evidence, introducing sentinel tasks with known answers into the workflow, and conducting retrospective audits comparing agent summaries against raw logs. The paper warns that these measures must be balanced against the risk of workplace surveillance.
The fourth involves organizational practices: periodic AI-free exercises to maintain core competencies, training on the distinction between fluency and correctness, mandatory breaks and rotations, assigning oversight roles strictly to those with domain expertise, separating supervisors from beneficiaries of the approval, and structuring incentives so as not to penalize thoroughness.
A final criterion, mine rather than the paper's: grant autonomy not based on how convincing the agent appears, but by asking whether the action is reversible, whether harm can be detected immediately, whether the approver possesses domain expertise and time to verify, and whether the system logs its actions in an auditable manner. These are engineering common sense, not experimental results.
Objections and Open Questions
The paper addresses three objections. First: the effects are exaggerated and people will adapt, as occurred with search engines. The authors respond that search engines return documents to be evaluated, whereas agents execute actions to be authorized, and that agent capabilities advance faster than cognitive adaptation research. However, adaptation remains possible, and the paper does not disprove it experimentally, as it conducts no experiments. Second: better tools and transparency will suffice. These are necessary, the authors argue, but insufficient, because explanations rely on cognitive faculties degraded by agent use and can themselves be flawed or act as anchors. My article on Cyborgs includes an example: a consultant asks GPT-4 to validate their recommendation, receives plausible confirmation, and accepts an incorrect answer. Third: alignment will make oversight redundant. The authors counter that this relies on unproven future capabilities and that training models on human preferences assumes those preferences measure control quality, whereas people tend to prefer fluent, agreeable responses.
Two additional observations of my own. Not all delegation leads to loss: freed time can be reinvested in analysis and learning, or converted into higher volume per worker, and available data cannot yet tell us which outcome dominates. Furthermore, the paper does not pit humans against machines: its subject is the socio-technical system encompassing both, alongside interfaces, goals, and incentives.
What does real-world practice show? In DigitalOcean's Currents report from February 2026, discussed in AI Agents Are Working. And the Math Checks Out, only 10% of respondents have fully autonomous agents in production, 40% require human review for everything, and 58% cite human approval points as their primary safety measure. The sample—around 1,100 self-reported responses from developers, tech leads, and founders—does not represent all work, but suggests that human approvers are currently the primary safety barrier in place, precisely the target of the position paper. The paper observes that this burden falls almost entirely on end users. Whether the signature serves to prevent errors or merely to assign blame after the fact is something neither the paper nor I can answer: it is a question each organization must put to its own processes.
AI agents are not necessarily forcing us out of the loop, but they do so whenever we mistake a signature for control, a log for understanding, or an approval rate for safety. Building capable agents is not enough: we need organizations and interfaces that preserve our ability to stop, correct, and understand them. In every workflow, one question remains: who answers for the final decision, and what tools do they have to earn that responsibility?