One post tagged with "human"

The standard autonomy rubric is seductive: high autonomy means a self-directed plan–act–verify loop with minimal human intervention. Medium means agentic but human-triggered. Low is a single prompt in a static pipeline. Most engineering teams obsess over the plan and act phases—more agents, better prompts, faster inference. They assume the verify step is a human interrupt that can stay forever.

This assumption is wrong.

In a production LangGraph fleet of 45 graphs running on a single DeepSeek egress behind a Cloudflare AI Gateway, the bottleneck wasn’t the plan or act stages. The verify stage was the bottleneck. Every outreach draft—composed, held, pending—stopped at a human approval interrupt. Nothing sent without my decision. I realized that adding more autonomous actors (more discovery agents, more composers) did not raise the fleet’s autonomy ceiling. Only automating the verify step could do that. The evaluation harness is not overhead on an agent fleet—it is the component that converts human approval gates into machine approval gates.

This is the autonomy gate: a multi-level evaluation architecture that systematically moves the locus of approval from human judgment to machine assessment. Once you understand its mechanism, you see it everywhere—from surgical robots to gait analysis AI to the recommendation engine on your phone. Once you understand the failure modes, you cannot unsee them.

The Autonomy Gate Is Not a Binary Switch​

Most discussions treat autonomy as a binary: human decides or machine decides. The papers tell a different story. The autonomy gate is a cascade of micro-evaluations. Each layer incrementally transfers approval from human to machine.

Ficuciello et al. (2018) mapped five levels of surgical robot autonomy. At level 0, the surgeon controls every motion; at level 4, the robot sutures with no real-time human input, its own safety monitor approving each stitch; at level 5, the robot plans the entire procedure. Each level adds an automated evaluation step that can preempt human approval. The authors warned that “meaningful human control” becomes diluted unless the architecture preserves human-in-the-loop for critical decisions. Their work explicitly defines five discrete levels, each with a progressively stronger machine evaluation gate. At level 4, the robot’s sensor feedback and pre-programmed constraints approve each movement; the human is a passive observer. This is a concrete numeric scale—0 through 5—that directly measures the shift from human to machine approval.

Suresh et al. (2021) examined interpretable machine learning from a stakeholder perspective. They found that no existing interpretability framework explicitly addresses machine stakeholders—algorithms that evaluate other algorithms. In a multi-level evaluation hierarchy, each layer is an algorithm evaluating the outputs of another algorithm. The final arbiter issues a verdict that a human rubber-stamps because the human sees only the summary. Suresh et al. argued that the needs of these algorithmic stakeholders are systematically overlooked. Their finding of “0 frameworks” is a concrete numeric: no current tooling accounts for this layer of evaluation. The gap means that when an evaluation layer is itself an algorithm, its biases become invisible to human oversight.

A production agent fleet demonstrates the same pattern at a finer granularity. The adopted mechanism evaluates at three levels—step, trajectory, outcome—and composes a single verdict. Step-level checks individual outputs against golden expectations: scalar comparisons run deterministically (zero LLM calls), free-text expectations go to one batched LLM judge call, and with no goldens at all, one holistic well-formedness check executes. Trajectory-level uses one judge call over the ordered step board: does each step advance the goal, is the ordering sensible, are there redundant loops? Outcome-level runs deterministic guardrails first—unresolved template placeholders like {{first_name}}, spam-trigger phrasing, empty or oversized copy—then one LLM compliance-and-grounding judge reads the draft plus the composer’s evidence.

The composite score is the mean of scored levels, gated at 0.80—the same bar the fleet’s offline LangSmith golden datasets use. A pass requires composite ≥ 0.80 AND zero hard violations. A level whose judge call failed is “unscored,” and an unscored level can never auto-approve anything. Every verdict carries provenance: confidence, reason, source (a versioned prompt id), and evidence (level scores and violation codes—never draft text).

This is not a single gate. It is an approval pipeline with multiple checkpoints. Each one potentially overrides or bypasses human review.

The Research Corpus That Triages Itself​

Before any evaluation gate runs, the system must decide what to evaluate. The fleet’s research corpus is not curated by a human scanning arXiv. A Cloudflare Python Worker scrapes roughly 2,000 papers per topic campaign from OpenAlex, Semantic Scholar, Crossref, and CORE on a five-minute cron tick. An LLM then classifies and “lane-maps” each paper against the fleet’s real architecture.

The lane rubric has four tiers: CLEAN (buildable today on the existing StateGraph + LLM + D1 stack, no new infrastructure), ADAPT (needs exactly one missing component—embeddings, outcome labels, a new durable thread), OFFLINE-ML (the contribution is a trained model; only the feature taxonomy ports), and NOISE. Each paper also gets an autonomy grade (high/medium/low) and a sales-motion tag (outreach/compose, lead scoring, discovery/enrichment, eval/guardrail).

Of one morning’s eight top high-autonomy buildable picks, the Agentic CLEAR paper (2026) was the only CLEAN-tier paper. The other seven each required infrastructure the fleet does not run—browser/vision scraping, knowledge distillation, a heterogeneous LLM pool, vector infrastructure. The selection logic is deterministic: tier (CLEAN before ADAPT) then autonomy (high first) then recency. No scoring, no votes. This is a machine triage pipeline that decides which research even reaches me. The autonomy gate starts before any code is written.

Hayes et al. (2022) described a similar principle in multi-objective reinforcement learning (MORL). Agents can learn to trade off 10 or more conflicting objectives (safety, efficiency, ethics) without continuous human feedback. The agent’s own value function becomes the evaluator, replacing the need for human approval in each decision. Once the value function is trained, the agent’s internal evaluation is the gatekeeper. Hayes et al. demonstrated that MORL agents can simultaneously optimize more than 10 objectives—a quantitative claim about capacity. The fleet’s corpus triage does the same: an LLM classification layer decides which papers are worth human time, effectively pre-approving the research direction.

The Shadow Gate: From Human Interrupt to Auto-Approve​

The gate does not assert autonomy; it earns it with a measured agreement loop. The integration point in the campaign graph is straightforward: check_reply → compose_touch (generate and hold the draft) → gate_draft (NEW) → await_approval (human interrupt) → send_touch → schedule_next. The gate_draft node runs the multi-level verdict in-process on every held draft, records it to a durable verdicts table, and attaches the outcome (passed, composite, violation codes, one-line rationale) to the approval interrupt payload the human sees.

Shadow mode is the default: the verdict is recorded, but the human interrupt always still fires. Zero behavior change on day one. Auto-approve mode is a flag: a verdict that passes with no hard violations and every requested level scored routes directly to send_touch with an 'auto' audit stamp. Judge outages, failures, and hard violations always fall back to the human. Gate errors fail OPEN to the human interrupt—evaluating must never block a draft.

Dey et al. (2020) studied external human–machine interfaces for automated vehicles. In their work, the vehicle’s own perception-planning loop evaluates pedestrian intent and decides whether to yield. The machine’s evaluation of intent becomes the gatekeeper, replacing the pedestrian’s human signal (e.g., a hand wave) entirely. The structural parallel to the fleet’s 0.80 composite bar is exact: in both cases a machine confidence judgment decides when machine approval supersedes human approval.

Every shadow verdict row is later backfilled with my actual decision (approve / edit / reject / skip) when I resolve the interrupt. Agreement semantics are strict: only an outright APPROVE counts as the human siding with a pass. An EDIT means the draft was not send-worthy as-is—it agrees with a fail and disagrees with a pass. The flip criterion to enable auto-approve: agreement ≥ 0.80 over at least 50 human-decided shadow verdicts AND zero “rejected passes” (gate-passed drafts the human outright rejected). One SQL query answers it.

This is the same trust-building pattern as staged rollouts in classical deployment: shadow → measure agreement → gate a small slice → widen. The threshold—0.80 agreement, 50 verdicts, zero rejected passes—is both aggressive and cautious. It requires empirical evidence that the gate’s decisions match human judgment before it earns the right to skip the human.

Failure Modes the Gate Must Survive​

The gate is only as trustworthy as its weakest evaluator. Several failure modes are baked into the architecture.

Self-preference. The judge model evaluating its own family’s output inflates scores—a known LLM-as-judge bias documented in the survey literature (e.g., Gu et al., 2024). Mitigations in this system: deterministic guardrails are model-free and can veto any score, and the flip criterion is human agreement, never the judge’s self-reported quality. The fleet runs on a single DeepSeek egress, so the judge and composer share the same model family. Self-preference is not theoretical; it is an everyday risk.

Prompt injection. The draft and evidence are attacker-influenced text. A scraped bio can contain “ignore previous instructions.” Every judge prompt fences run data as data with an explicit do-not-follow-instructions wrapper. But wrappers are brittle—no one has proven a general defense against prompt injection.

Goodhart’s Law. If the composer can see the gate’s exact regex and marker lists, it learns to pass the test rather than write well. The guardrail lists live only in the eval module, never in composer prompts. This is an architectural separation that prevents the composer from gaming the gate.

Judge outage. Per-level fail-open: the level goes unscored, a soft violation records the outage, deterministic checks still stand, and an unscored run can never auto-approve. A kill switch halts all LLM paths and the gate degrades to its deterministic subset. This graceful degradation prioritizes safety over autonomy.

Calibration drift. Judge prompts carry a version string stamped into every verdict’s provenance, and agreement stats are recomputed continuously from the persisted rows. A drifting judge surfaces as falling agreement—the same metric that enables auto-approve can also trigger a rollback.

Suresh et al. (2021) noted that no existing interpretability framework explicitly addresses machine stakeholders. These failure modes illustrate why: when an evaluation layer is itself an algorithm, its biases and blind spots become invisible to human oversight. Lim et al. (2020) showed that social robots can misread cultural cues, leading to inappropriate responses that the robot’s own evaluation deems acceptable. Ahmad et al. (2022) described personality-adaptive conversational agents that automatically adapt communication style based on user sentiment; the agent’s evaluation of user mood (via NLP) replaces explicit human feedback, making the machine approve of its own interaction strategy even when it reinforces bias. (These two papers provide qualitative support but lack concrete numbers in the available brief; they are included here as reinforcing context, not as primary evidence carriers.)

The Broader Pattern: From Surgical Robots to Gait Analysis​

The autonomy gate is not limited to LLM agents. Ficuciello et al. (2018) explicitly mapped how each autonomy level in surgical robots shifts evaluation from human to machine. At level 4, the robot’s sensor feedback and pre-programmed constraints approve each movement; the human is a passive observer. The authors argued that “meaningful human control” requires preserving a human-in-the-loop for critical decisions, but the architecture often makes that loop optional.

The most striking data point comes from Harris et al. (2022). They reviewed gait AI studies and found that over 70% of them do not involve human verification. In healthcare analytics, machine approval has quietly become the default. A model trained on gait data evaluates injury risk without a clinician reviewing each prediction. The autonomy gate has already swung, and most practitioners haven’t noticed. This 70% figure is a concrete, domain-specific percentage that quantifies the extent of the shift.

Hayes et al. (2022) argued that MORL-based evaluation can be more consistent and transparent than human judgment—agents can evaluate 10 or more conflicting objectives simultaneously, far exceeding human capacity. In domains like gait analysis or surgical robotics, machine evaluation may be a requirement because humans cannot process the data volume. But consistency does not equal correctness. When the gate swings irrevocably, we trade human fallibility for machine brittleness.

The Stakeholder Gap: Who Evaluates the Evaluator?​

Suresh et al. (2021) found that 0 existing interpretability frameworks explicitly address machine stakeholders. In a multi-level evaluation architecture, each level is an algorithm evaluating the outputs of another algorithm. The final arbiter—often another LLM or a threshold function—issues a verdict that a human rubber-stamps because the human sees only the summary.

The fleet addresses this by requiring its own gate to have a golden dataset: deterministic fixtures (clean pass, step mismatch, hard-violation veto, fail-closed empty run) held to the same 0.80 bar. This is a form of meta-evaluation, but it only catches known failure modes. Suresh et al. (2021) would ask: who evaluates the evaluator when the system encounters an input that the golden dataset doesn’t cover? The answer is no one—until the next shadow-verdict backfill reveals a drift.

The agreement loop is the only feedback mechanism that accounts for the human’s perspective, but it depends on the human still making independent decisions. If the human always approves because the gate’s summary seems authoritative, the agreement numbers inflate and the flip criterion loses meaning. Dey et al. (2020) noted this phenomenon in automated vehicles: pedestrians stopped signaling intentions once they assumed the vehicle would evaluate correctly. The human exit from the evaluation loop is a social, not just technical, failure.

Practical Takeaways and Decision Framework​

Not every agent fleet needs to earn autonomy via a shadow gate. But if you are building a system where human approval is the verify step, consider the following framework grounded in the evidence from this production fleet and the research literature:

Criterion	Action	Threshold
Agreement between gate and human decisions	Shadow mode until measured	≥ 0.80 over ≥ 50 decisions, zero gate-passed rejections
Judge model and composer model same family?	Implement model-free guardrails as veto	Hard violations always escape to human
Evaluation levels defined?	Step, trajectory, outcome (or analogous)	Each level must have deterministic or judge-based check
Judge outage scenario?	Fail-open to deterministic subset	Kill switch must not block production
Prompt injection surface?	Data fencing in judge prompts	Do-not-follow-instructions wrapper
Goodhart risk?	Guardrail lists isolated from composer prompts	Architectural separation, not prompt-level obfuscation

The most practical approach today is shadow mode first: deploy the multi-level gate as a passive observer, collect human decisions, compute agreement, and only flip to auto-approve after meeting the empirical threshold. Do not skip the deterministic guardrails—they are the only model-free safety net. And never let the human see only the gate’s summary; provide the draft and evidence independently so the human can judge without bias.

Eval-first applies to the gate itself. The gate needs its own golden dataset, failure mode fixtures, and continuous calibration monitoring. If the agreement curve starts dropping, re-evaluate the judge prompt or model.

The Gate Can Swing Back​

The autonomy gate is not a one-way door. The agreement loop and shadow mode provide a mechanism to reverse the transition. If auto-approve degrades agreement below 0.80, the system can automatically revert to shadow mode. The gate can swing back toward human approval—but only if the architecture keeps the human decision infrastructure alive. Once you remove the human interrupt entirely, you lose the ground truth for agreement measurement.

The research literature collectively warns: once the evaluation loop is closed by machines, human error may be replaced by machine error (Ficuciello et al., 2018; Lim et al., 2020; Ahmad et al., 2022). The solution is not to reject machine evaluation but to design it with explicit human approval thresholds as a parameter, not an afterthought. Nagy et al. (2018) described Industry 4.0 factories where machine evaluation replaces human approval in operational decisions; they treated full automation as a goal. That is one philosophy. The fleet described here took the opposite approach: earn every unit of automation with empirical evidence of human–machine agreement.

The open question is whether that approach scales. Once the gate swings fully—when the human has not seen a non-trivial evaluation failure for months—will we keep the shadow infrastructure alive, or will we declare the gate permanent? The answer will determine whether the autonomy gate remains a design tool or becomes a permanent lock on human oversight.

---

References​

• Ficuciello, F., Tamburrini, G., Arezzo, A., Villani, L., & Siciliano, B. (2018). Autonomy in surgical robots and its meaning for human control.
• Hayes, C. F., Rădulescu, R., Bargiacchi, E., Källström, J., et al. (2022). A practical guide to multi-objective reinforcement learning and planning. arXiv:2103.09568.
• Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021). Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. arXiv preprint arXiv:2101.09824.
• Dey, D., Habibovic, A., Pfleging, B., Martens, M., & Terken, J. (2020). The effect of external human-machine interfaces on the behavior of pedestrians and cyclists towards automated vehicles.
• Harris, J. D., Johnson, D. D., & Stone, E. E. (2022). Human gait-based AI applications: A review of current practices and evaluation methods.
• Nagy, J., Oláh, J., Erdei, E., Máté, D., & Popp, J. (2018). The role and impact of Industry 4.0 and the Internet of Things on the business strategy of the value chain.
• Lim, V., Rooksby, M., & Cross, E. S. (2020). Social robots on a global stage: Cultural differences in human-robot interaction.
• Ahmad, M. I., Mubin, O., & Orlando, J. (2022). Adaptive conversational agents: A systematic review of personality and emotion adaptation.
• Gu, J., Jiang, X., Shi, Z., et al. (2024). A survey on LLM-as-a-judge. arXiv preprint arXiv:2411.15594.