Human in the loop is not a get out of jail free card

I’m old enough to remember when half the advantage of automating a process was removing human error. So I’m increasingly suspicious when a discussion of an AI system’s risks ends with “there’s a human in the loop.”

In my experience, that phrase too often substitutes for the details of how the system will work. Someone approves the recommendation, so the risks have been addressed. If the recommendation turns out to be wrong, that person should have caught it.

But putting a person at the end of a workflow creates a design problem of its own. It can make human attention the bottleneck on the speed and scale promised by automation, while delivering far less of the improvement in quality that is supposed to justify the slowdown.

For many business processes, the promise of automation is straightforward: do the work faster, more reliably, and at greater volume. Human review can be a sensible tradeoff against those goals. The mistake is treating its cost as small and its benefit as automatic.

Consider a system that reviews data, makes a recommendation, and waits for someone to approve the resulting action. The model might produce recommendations in milliseconds. If every recommendation requires a minute of human attention, the system’s capacity still depends on how many minutes of review the organization can provide. Automating the analysis may save substantial work, but the review step sets a limit on how far those savings can scale.

And that minute may be an optimistic assumption. Reading an answer and verifying it are different amounts of work. A reviewer may need to retrieve the underlying evidence, check whether it supports the conclusion, and reconstruct assumptions the recommendation leaves unstated. In some cases, much of the original analysis has to be repeated before the reviewer can confidently approve it.

So perhaps that is the deal: accept the bottleneck in exchange for better decisions. The model does the initial work, the human catches its mistakes, and the combination is more reliable than either alone.

That intuition is particularly appealing if you work in machine learning. Ensemble methods give us good reasons to look for gains from combining imperfect models. But those gains depend on which errors the components make and how their outputs are combined. Adding another decision maker does not, by itself, guarantee an improvement. A reviewer looking at an AI recommendation is also responding to that recommendation, rather than supplying a fully independent judgment.

The empirical results are a useful challenge to this intuition. In a 2024 meta-analysis of 106 experimental studies, Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone found that human–AI combinations performed worse, on average, than the better of the human or AI alone. They found performance losses in decision tasks and greater gains in content creation. That does not mean human review generally makes people worse than they would be without AI. It means the combination cannot be assumed to outperform its strongest component. The quality benefit has to be demonstrated.[1]

There are several reasons I would be skeptical of assuming that benefit in a system design.

First, automation changes the job the human is doing. Reviewing a completed recommendation is different from developing one. The person who did the original work encountered the ambiguities, considered alternatives, and learned which details mattered. The reviewer has to acquire enough of that understanding to judge an answer that already looks finished.

My suspicion is that ownership matters here too. When I have invested effort in a recommendation, I feel more responsible for defending it and more compelled to check its details. Approving someone else’s answer can create a much thinner connection to the result. Though not guaranteed, the analogy of craftsmen vs assembly line workers feels relevant to considering the tradeoffs between speed and quality when dividing tasks into smaller steps.

Second, repeated correctness changes the reviewer’s experience of the task. Imagine reviewing recommendations that are right 99% of the time. Most careful checks produce no correction. Other work is waiting. Spending less time on each recommendation becomes an understandable response to what the reviewer has learned about the system and where their time could be best spent.

Research on automation complacency and automation bias corroborates this feeling. Parasuraman and Manzey’s review identifies attention as central to these effects, including situations where other tasks compete with monitoring for the operator’s attention.[2] When designing systems, we need to take into consideration how people actually behave, inevitably with competing demands, rather than as theoretically rational actors in a vacuum.

Third, the ability to intervene may itself deteriorate. Someone who mainly approves recommendations gets less practice developing them independently. That creates a possible dependency: the system reduces the exercise of the judgment it still relies on when something goes wrong. Lisanne Bainbridge described this problem in “Ironies of Automation” in 1983, long before the current generation of AI systems. Automating routine work can leave people responsible for exceptional situations while making it harder to maintain the skills those situations require.[3]

Routing only exceptions to humans changes the problem again. It can reduce the approval bottleneck, but now the remaining cases may be systematically harder. Automating half the cases does not imply saving half the labor if those cases were the easy ones. Longer handling times and lower accuracy on the remaining cases may reflect the changed caseload before any loss of proficiency occurs. Any system design has to account for both possibilities.

These effects make the speed and quality problems reinforce one another. Thorough verification is expensive. Pressure to achieve the promised throughput encourages lighter review. Lighter review weakens the safeguard that justified retaining the human approval step. A system can end up paying for review without getting the protection its architects assumed.

None of this makes human judgment dispensable. People may have relevant context the system lacks, recognize that the recommendation answers the wrong question, or make tradeoffs that properly belong to an accountable person. Those are good reasons to involve them. But the design should specify what judgment they are contributing and give them the information, time, and authority to exercise it.

That means being concrete about the control. What errors is the reviewer expected to catch? How much work does detecting them require? Can the reviewer handle the expected volume? What happens when they disagree? Most importantly, does the review actually catch consequential errors under realistic workloads, including after people have become familiar with the system?

Some checks can be handled by deterministic rules. Some cases can be routed for deeper review. Sampling can help measure performance, although it does not protect every unreviewed action. Confidence filters may help if they reliably identify the errors that matter. Each is a design choice with a benefit to establish and a limit to understand.

There is also a question of responsibility. Madeleine Clare Elish calls attention to “moral crumple zones”: people who absorb responsibility for failures of automated systems despite having limited control over their behavior.[4] A required approval can create the appearance of human control without providing its substance.

If human review is a safeguard in your system, its effectiveness is part of your design responsibility. You chose the volume, the interface, the evidence available, and the effort required to challenge the answer. Your responsibility does not end at the seam between human and machine. Predictable limits on human attention and judgment belong in that design, as a true system encompasses input and outcome, not just the actions of the technology.

“A human approved it” tells us that an approval occurred. The people who built and deployed the system still need to explain why the promised benefits will be realized without assuming that human can wave a magic wand.

Sources

1. Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone (2024), “When combinations of humans and AI are useful: A systematic review and meta-analysis.” The comparison is against the better standalone performer, not necessarily against humans alone. The studies covered were published from January 2020 through June 2023; this is not an evaluation of every current AI workflow.
2. Raja Parasuraman and Dietrich H. Manzey (2010), “Complacency and bias in human use of automation: An attentional integration.” Supports the discussion of attention, monitoring, and automation bias; it does not establish a universal error-rate threshold at which review fails.
3. Lisanne Bainbridge (1983), “Ironies of automation.”90046-8) Provides the foundation for the argument about maintaining human skill and readiness when routine work is automated.
4. Madeleine Clare Elish (2019), “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction.” Addresses the attribution of responsibility to humans with limited control in automated systems.