What is the difference between human-in-the-loop, on-the-loop, and out-of-the-loop?

In-the-loop means the AI proposes and a person approves before anything happens, so no action occurs without a human decision. On-the-loop means the system acts on its own while a person supervises and can intervene, usually through monitoring and alerts. Out-of-the-loop means the system acts and no one reviews individual actions; oversight, if it exists, is after-the-fact auditing. They differ in where the authority to act sits, which has little to do with how capable the model is.

How do I choose which level to use?

Ask what one unreviewed mistake would cost you, then work backwards. Severity comes first: a misrouted support reply is cheap and constant, which argues for less human involvement, while a payment, a contract change or a rejection letter carries a bill you would rather not pay unreviewed. Reversibility discounts the severity, since a mistake you can cheaply claw back is a smaller problem than one that stands. Frequency sets the budget: if you cannot afford to look at every case, you have already chosen a looser level whether you admit it or not.

Does adding a human always make an AI system safer?

No. A reviewer who approves everything adds latency without adding judgement, and can make outcomes worse by lending authority to decisions nobody examined. Oversight only works when the reviewer has the time, the information, and the standing to say no. If any of those three is missing, you pay for a checkpoint that does not protect you.

What is automation bias?

It is the tendency to defer to a machine's recommendation even when you have the evidence to question it. It matters here because it can turn an in-the-loop design into an out-of-the-loop one without anyone noticing: the approval step still exists on the org chart, but the human is no longer the one deciding. This is why review rates that look healthy can hide an oversight process that stopped functioning.

How is this different from just deciding what to automate?

Automation asks whether a task should be done by a machine. This asks who holds the authority to act once the machine can do it. The same task can sit at all three levels depending on the stakes: an AI can draft an email that you send, send routine emails while you watch the queue, or send confirmations nobody reads. The work is identical in all three cases, but the control structure differs.

What is the most common mistake teams make with these designs?

The most common mistake is drifting between levels without deciding to. A system launches with a person approving every action, volume grows, approvals become rubber stamps, and nobody records that the design changed. The system is now effectively out of the loop while every document still describes a human checkpoint. Failures then get blamed on the model when the cause is an oversight step that stopped working.

How does this apply to hiring and job applications?

Many hiring pipelines now mix all three. Formatting and keyword parsing typically run unattended, ranking and scoring often run with a recruiter supervising, and final decisions stay with people. For a candidate, the practical consequence is that the earliest stages are the least likely to have someone reading carefully, so material that only makes sense to a sympathetic human reader can be filtered before any human sees it.

Should AI agents ever run fully out of the loop?

Yes, for actions that are frequent, cheap when wrong, and easily corrected, where inserting a person would cost more than the errors do. Spam filtering is the classic case. Teams get into trouble by extending that setting to decisions they could not afford to get wrong unreviewed, then discovering the bill only after enough of them have accumulated.

Human In, On, or Out of the Loop: Where People Actually Belong in an AI System

Teams building with AI end up arguing about the same thing sooner or later, often without naming it: how much should a person be involved, and involved where. The argument usually gets framed as a safety question, or a trust question. Underneath both is a more specific question: who holds the authority to act. Fields with no connection to machine learning have been studying that for decades.

The short version: there are three workable places to put a person. In the loop, the system proposes and a human approves before anything happens. On the loop, the system acts and a human supervises, stepping in on exceptions. Out of the loop, the system acts and nobody reviews the individual decisions. Choose by what one unreviewed mistake would cost you. After that, the failure to watch for is drifting between levels without noticing.

What do in, on, and out of the loop mean?

The distinction turns on where the decision right sits, and much less on how much automation exists.

Human in the loop. The system produces a recommendation and stops there. Nothing executes until a person approves it. This is the most protective arrangement available and the most expensive one, because every action buys a checkpoint and pays for it twice, once in latency and once in the reviewer's attention, which is the scarcer of the two.

Human on the loop. The system acts by itself, and a person supervises through dashboards, alerts, and sampling, stepping in when something looks wrong. The system keeps running when the supervisor looks away, which is both why teams choose this level and where its risk comes from.

Human out of the loop. The system acts and no one reviews individual actions. Oversight, where it exists at all, is auditing after the fact. Spam filtering lives here for a good reason: the decisions are constant, individually cheap, and easy to reverse.

	Who acts	Who decides	What it costs	Best fit
In the loop	System proposes	Person, every time	Latency and reviewer attention	Too costly to get wrong unreviewed
On the loop	System	Person, on exceptions	Monitoring that has to work	Real stakes, but speed matters more than certainty
Out of the loop	System	Nobody, per action	Errors you only find later	Cheap when wrong, frequent enough that checking costs more

The same task can sit at any of the three levels. An AI can draft an email that you send, send routine emails while you watch the queue, or send confirmations nobody ever reads. The work is identical in all three cases, but the control structure is completely different, and so is who is accountable when it goes wrong.

Why is this an economics problem before it is an engineering problem?

I did my PhD in economics, and this framework was familiar long before I built AI systems for a living. It is the delegation problem, which industrial organization has picked apart for half a century.

Industrial organization asks where activities should sit: inside the firm or bought from the market, decided by the manager or by the person on the floor. The core insight is that authority has a cost structure. Delegating gets you speed and lets the decision use information the centre does not have. Retaining authority gets you control and consistency, and you pay for it in bottlenecks and in the attention of whoever must approve.

That literature also produced the distinction that matters most here: formal authority versus real authority. Formal authority is who signs off. Real authority is who determines the outcome. They come apart whenever the person with formal authority lacks the time or information to dissent meaningfully. A manager who approves every proposal because there are three hundred of them holds formal authority and no real authority at all.

Read the AI oversight debate through that lens and the vocabulary changes. Automation bias, the well-documented tendency to defer to the machine's recommendation, gets treated as a quirk of psychology. Seen this way it looks more like an org design problem: real authority migrated to the system while formal authority stayed with the human. The org chart still says in-the-loop while the decisions are effectively made out of the loop, and because nobody wrote that change down, nobody corrects it.

So whenever someone asks "should a human review this?", follow up with a narrower question: does the reviewer have the time to look, the information to judge, and the standing to say no when the answer is no? Miss any one of those and you are paying for a checkpoint you do not have.

What does quant trading already know about this?

There is a second field that ran this experiment for decades with real money, and it has a name for the argument: discretionary versus systematic.

A discretionary trader decides each position. A systematic strategy encodes the rules and executes them without asking. That is in-the-loop versus out-of-the-loop, and the intermediate case (systematic execution with human overrides and kill switches) is on-the-loop.

That argument settled two things that carry over to AI.

The first is that the kind of decision determines which approach works, however skilled the decider is. Systematic wins where an edge is small, repeatable and has to be applied consistently at volume, because a human intervening there adds variance without adding information. Discretion earns its cost where situations are novel and the relevant information has never been encoded. Experienced practitioners rarely claim one approach is better everywhere; they argue about which regime they are in.

The second is more uncomfortable and transfers directly: the dangerous configuration is the undeclared hybrid. A systematic book with frequent discretionary overrides is neither systematic nor discretionary. Nobody has characterised how it behaves, so its results cannot be attributed. If you cannot tell whether a drawdown came from the model or from the overrides, you cannot fix it.

That is the same failure as an AI agent nominally supervised by a human who no longer reads the alerts. In both cases, the design on paper and the design in production had drifted apart without anyone noticing, so every diagnosis afterwards aimed at the wrong target.

How do you choose a level, then?

Start with the cost of being wrong when nobody checked. Severity is the first term: how much damage does one bad action do. Reversibility then discounts it, because a mistake you can claw back cheaply is a smaller bill than one that stands. Frequency sets the budget on the other side, since every check has to be paid for out of someone's attention.

Reversibility on its own is a poor bar, and it is the one many teams reach for first. An action can be permanent and trivial, and it can be technically undoable and still ruin the week by the time anyone notices. The question to ask is whether you could live with this going wrong unreviewed.

Frequency deserves more weight than it usually gets, because it sets the honest budget for oversight. If a thousand actions a day need approval and one reviewer exists, in-the-loop is a fiction that will resolve itself into rubber-stamping within a month. It is better to choose on-the-loop deliberately, with alerting and sampling that work, than to choose in-the-loop and let it decay.

Three questions worth answering before shipping any agentic system:

What is the worst action this can take, and could we afford it going wrong unreviewed? If the answer is no, that action belongs in the loop whatever the rest of the system does.
What would make the supervisor intervene, and would they see it in time? On-the-loop without a working answer here is out-of-the-loop with extra staffing.
How will we notice if the level changes? Approval rates near a hundred percent, alerts nobody acknowledges, and overrides that quietly stop happening are all signals that the design moved.

Treat the three as settings rather than stages. Moving an action outward is not an achievement on its own, and keeping it in is not a virtue either. Different actions inside the same product can stay at different levels permanently.

What does this mean in a job application?

Hiring is one of the places where these three levels now sit side by side, and where many candidates assume more human attention exists than there is.

In many hiring pipelines, parsing and formatting checks run unattended. Ranking and scoring commonly run with a recruiter supervising a queue rather than reading every profile. Interviews and offers are still handled by people. That mix has a practical consequence: the earliest stage of your application is the least likely to have someone reading it carefully, so anything that requires a sympathetic reader to interpret it (an unusual title, a career break you would explain in one sentence, a project whose relevance is obvious only if someone asks) is being evaluated where nobody is asking.

So make your material easy to understand without a human advocate present: state the role you are targeting plainly, make the relevant work visible without inference, and address the gap before anyone has to wonder about it.

The same reasoning is behind how we build PokeBot: scoring and feedback run automatically because they are frequent and reversible, and the decisions that matter (what to target, what to change, what to say) stay with the person whose career it is.

What is the hardest part to get right?

The three levels are easy to learn and easy to misuse, because most of the work comes after you classify a system: being honest about which level you are operating at, and noticing when that stops being true.

Economics has a name for the gap between the person who signs and the person who decides, and trading has one for the book that is neither systematic nor discretionary. Both fields learned the same lesson at some expense: oversight that exists on paper and not in practice is worse than no oversight at all, because it gives people confidence in decisions nobody reviewed.