What is the difference between human-in-the-loop, on-the-loop, and out-of-the-loop?

In-the-loop means the AI proposes and a person approves before anything happens, so no action occurs without a human decision. On-the-loop means the system acts on its own while a person supervises and can intervene, usually through monitoring and alerts. Out-of-the-loop means the system acts and no one reviews individual actions; oversight, if it exists, is after-the-fact auditing. They differ in where the authority to act sits, which has little to do with how capable the model is.

How do I choose which level to use?

Ask what one unreviewed mistake would cost you, then work backwards. Severity comes first: a misrouted support reply is cheap and constant, which argues for less human involvement, while a payment, a contract change or a rejection letter carries a bill you would rather not pay unreviewed. Reversibility discounts the severity, since a mistake you can cheaply claw back is a smaller problem than one that stands. Frequency sets the budget: if you cannot afford to look at every case, you have already chosen a looser level whether you admit it or not.

Does adding a human always make an AI system safer?

No. A reviewer who approves everything adds latency without adding judgement, and can make outcomes worse by lending authority to decisions nobody actually examined. Oversight only works when the reviewer has the time, the information, and the standing to say no. If any of those three is missing, you have the cost of a checkpoint without the protection of one.

What is automation bias?

It is the tendency to defer to a machine's recommendation even when you have the evidence to question it. It matters here because it quietly converts an in-the-loop design into an out-of-the-loop one: the approval step still exists on the org chart, but the human is no longer the one deciding. This is why review rates that look healthy can hide an oversight process that stopped functioning.

How is this different from just deciding what to automate?

Automation asks whether a task should be done by a machine. This asks who holds the authority to act once the machine can do it. The same task can sit at all three levels depending on the stakes: an AI can draft an email that you send, send routine emails while you watch the queue, or send confirmations nobody reads. The work is identical. The control structure is not.

What is the most common mistake teams make with these designs?

Drifting between levels without deciding to. A system launches with a person approving every action, volume grows, approvals become rubber stamps, and nobody records that the design changed. The system is now effectively out of the loop while every document still describes a human checkpoint. Failures then get attributed to the model when the actual cause is an oversight step that stopped being real.

How does this apply to hiring and job applications?

Most hiring pipelines now mix all three. Formatting and keyword parsing typically run unattended, ranking and scoring often run with a recruiter supervising, and final decisions stay with people. For a candidate, the practical consequence is that the earliest stages are the least likely to have someone reading carefully, so material that only makes sense to a sympathetic human reader can be filtered before any human sees it.

Should AI agents ever run fully out of the loop?

Yes, for actions that are frequent, cheap when wrong, and easily corrected, where inserting a person would cost more than the errors do. Spam filtering is the classic case. Teams get into trouble by extending that setting to decisions they could not actually afford to get wrong unreviewed, then discovering the bill only after enough of them have accumulated.

Human In, On, or Out of the Loop: Where People Actually Belong in an AI System

Every team building with AI eventually argues about the same thing, usually without naming it: how much should a person be involved, and involved where. The argument usually gets framed as a safety question, or a trust question. Underneath it sits something more specific: who holds the authority to act. Fields with no connection to machine learning have been studying that for decades.

The short version: there are three workable places to put a person. In the loop, where the system proposes and a human approves before anything happens. On the loop, where the system acts and a human supervises, intervening on exceptions. Out of the loop, where the system acts and nobody reviews the individual decisions. Choose by what one unreviewed mistake would cost you. Then watch for the real failure, which is drifting between levels without noticing.

What do in, on, and out of the loop actually mean?

The distinction turns on where the decision right sits. How much automation exists barely enters into it.

Human in the loop. The system produces a recommendation and stops there. Nothing executes until a person approves it. The machine proposes; the person disposes. This is the most protective arrangement available and the most expensive one, because every single action buys a checkpoint and pays for it twice, once in latency and once in the reviewer's attention, which is the scarcer of the two.

Human on the loop. The system acts by itself, and a person supervises through dashboards, alerts, and sampling, stepping in when something looks wrong. The system keeps running when the supervisor looks away — that is the point, and also the risk.

Human out of the loop. The system acts and no one reviews individual actions. Oversight, where it exists at all, is auditing after the fact. Spam filtering lives here for a good reason: the decisions are constant, individually cheap, and easy to reverse.

	Who acts	Who decides	What it costs	Best fit
In the loop	System proposes	Person, every time	Latency and reviewer attention	Too costly to get wrong unreviewed
On the loop	System	Person, on exceptions	Monitoring that has to actually work	Real stakes, but speed matters more than certainty
Out of the loop	System	Nobody, per action	Errors you only find later	Cheap when wrong, frequent enough that checking costs more

A useful clarifier: the same task can sit at any of the three levels. An AI can draft an email that you send, send routine emails while you watch the queue, or send confirmations nobody ever reads. The work is identical. The control structure is completely different, and so is who is accountable when it goes wrong.

Why is this an economics problem before it is an engineering problem?

I did my PhD in economics, and this framework was familiar long before I built AI systems for a living — because it is the delegation problem, which industrial organization has picked apart for half a century.

Industrial organization asks where activities should sit: inside the firm or bought from the market, decided by the manager or by the person on the floor. The core insight is that authority has a cost structure. Delegating gets you speed and lets the decision use information the centre does not have. Retaining authority gets you control and consistency, and you pay for it in bottlenecks and in the attention of whoever must approve.

That literature also produced the distinction that matters most here: formal authority versus real authority. Formal authority is who signs off. Real authority is who actually determines the outcome. They come apart whenever the person with formal authority lacks the time or information to dissent meaningfully. A manager who approves every proposal because there are three hundred of them holds formal authority and no real authority at all.

Read the AI oversight debate through that lens and the vocabulary changes. Automation bias, the well-documented tendency to defer to the machine's recommendation, gets treated as a quirk of psychology. Seen this way it looks more like an org design problem: real authority migrated to the system while formal authority stayed with the human. The org chart says in-the-loop. The decisions say out-of-the-loop. Nobody wrote that change down, which is precisely why it survives.

So the practical question is narrower than "should a human review this?" Ask instead: does the reviewer have the time to look, the information to judge, and the standing to say no when the answer is no? Miss any one of those and you are paying for a checkpoint you do not have.

What does quant trading already know about this?

There is a second field that ran this experiment for decades with real money, and it has a name for the argument: discretionary versus systematic.

A discretionary trader decides each position. A systematic strategy encodes the rules and executes them without asking. This is exactly in-the-loop versus out-of-the-loop, with the intermediate case — systematic execution with human overrides and kill switches — sitting precisely where on-the-loop sits.

Two things that argument settled are worth importing.

The first is that the decision dictates the approach, and the decider's skill has little to do with it. Systematic wins where an edge is small, repeatable and has to be applied consistently at volume, because a human intervening there adds variance without adding information. Discretion earns its cost where situations are genuinely novel and the relevant information has never been encoded. Nobody sensible argues one is universally better. They argue about which regime they are in.

The second is more uncomfortable and transfers directly: the dangerous configuration is the undeclared hybrid. A systematic book with frequent discretionary overrides is not a systematic book, and it is not a discretionary one either. It is a system whose actual behaviour nobody has characterised, which means its results cannot be attributed. Was the drawdown the model or the overrides? You cannot answer, so you cannot fix it.

That is the same failure as an AI agent nominally supervised by a human who no longer reads the alerts. Supervision was never the problem. The design on paper and the design in production had quietly diverged, so every diagnosis afterwards aimed at the wrong target.

How do you choose a level, then?

Start with the cost of being wrong when nobody checked. Severity is the first term: how much damage does one bad action do. Reversibility then discounts it, because a mistake you can claw back cheaply is a smaller bill than one that stands. Frequency sets the budget on the other side, since every check has to be paid for out of someone's attention.

Reversibility on its own is a poor bar, and it is the one most teams reach for. An action can be permanent and trivial, and it can be technically undoable and still ruin the week by the time anyone notices. What you actually want to know is whether you could live with this going wrong unreviewed.

Frequency deserves more weight than it usually gets, because it sets the honest budget for oversight. If a thousand actions a day need approval and one reviewer exists, in-the-loop is a fiction that will resolve itself into rubber-stamping within a month. Choosing on-the-loop deliberately, with real alerting and real sampling, beats choosing in-the-loop and letting it decay.

Three questions worth answering before shipping any agentic system:

What is the worst action this can take, and could we afford it going wrong unreviewed? If the answer is no, that action belongs in the loop whatever the rest of the system does.
What would make the supervisor intervene, and would they see it in time? On-the-loop without a working answer here is out-of-the-loop with extra staffing.
How will we notice if the level changes? Approval rates near a hundred percent, alerts nobody acknowledges, and overrides that quietly stop happening are all signals that the design moved.

Treat the three as settings rather than stages. Moving outward is no achievement on its own, and staying in is no virtue either. Different actions inside the same product belong at different levels, permanently.

What does this mean in a job application?

Hiring is one of the places where these three levels now sit side by side, and where most candidates assume more human attention exists than actually does.

Parsing and formatting checks generally run unattended. Ranking and scoring commonly run with a recruiter supervising a queue rather than reading every profile. Interviews and offers remain firmly human. That mix has a practical consequence: the earliest stage of your application is the least likely to have someone reading it carefully, so anything that requires a sympathetic reader to interpret it — an unusual title, a career break you would explain in one sentence, a project whose relevance is obvious only if someone asks — is being evaluated where nobody is asking.

The move is to make the material legible without a human advocate present: the role you are targeting stated plainly, the relevant work visible without inference, the gap addressed before anyone has to wonder about it. You are writing for the loop you are actually in.

That is also the design principle behind how we build PokeBot: scoring and feedback run automatically because they are frequent and reversible, and the decisions that matter — what to target, what to change, what to say — stay with the person whose career it is.

The part worth remembering

The three levels are easy to learn and easy to misuse, because the interesting work starts after the classification. It lies in being honest about which one you are actually operating, and in noticing when that stops being true.

Economics has a name for the gap between the person who signs and the person who decides. Trading has a name for the book that is neither systematic nor discretionary. Both fields learned the same lesson at some expense: an oversight structure that exists on paper and not in practice is worse than no oversight at all, because it buys confidence nobody earned.