Computer-Use Agents for UI Verification: The Agent Explores, the Code Judges

In late June 2026, the best computer-use agent failed four of five tasks on the new OSWorld 2.0 benchmark. Claude Opus 4.8 with max thinking hit just 20.6% binary completion on 108 long-horizon tasks, each requiring a median 1.6 hours of human interaction and roughly 318 tool calls. GPT-5.5 plateaued near 13%. (OSWorld 2.0, arXiv:2606.29537, submitted 2026-06-28.)

That reset changed what I think verification agents should be. Most researchers saw a challenge: build better agents that operate other people’s software. I saw the inversion. A computer-use agent that wanders any UI is too unreliable for unsupervised verification. But a bounded agent that operates only your own UI, with deterministic assertions as the decider and an LLM as an optional explainer, works today. It costs near zero per run, catches real regressions, and never grades its own homework.

This post documents the lane I built for my own site: just over 400 lines of Playwright-driven verification, a BFS crawler for route discovery, a 64×64 visual diff, and a closed loop where the page publishing the results is itself under verification. I’ll show how it draws from foundational research — OSWorld 2.0, the Rico, PIXELHELP, the Open Agent Architecture — and where those papers fall short for verification on broken UIs.

The Inversion: Verification Agents vs. Assistant Agents

The dominant computer-use narrative is about assistants — agents that navigate other software on behalf of a human. The inference-time self-improvement thread, browser-use skill distillation (COLLEAGUE.SKILL, arXiv:2605.31264; online skill learning for web agents, arXiv:2606.04391), and black-box auditing all aim to make agents more capable on unfamiliar interfaces.

Verification of your own UI is the opposite problem. The human already knows what correct looks like. The challenge is not deciding what to do but proving that each commit hasn’t broken anything — without coupling tests to DOM structure or pixel-perfect baselines that flood CI with false alarms.

Traditional end-to-end testing relies on element selectors that break when a CSS class changes. Visual regression tools like Percy compare screenshots pixel-by-pixel and flag every sub-pixel anti-aliasing difference as a failure. Both approaches produce high false-positive rates. A computer-use agent perceives the UI visually, like a human, making it robust to DOM changes. But as OSWorld 2.0 shows, agents are unreliable — 20.6% is not CI-gate material.

The inversion has a simple rule: the agent explores, but a deterministic verifier judges. This mirrors the Open Agent Architecture (Martin, Cheyer & Moran, 1999), which coordinated distributed autonomous agents through a facilitator that brokered requests by matching them against each agent's advertised capabilities. In my lane, the Playwright navigator plays the coordinating role, but the verification module (pure asserts) and the reporter are independent. Each can be swapped without touching the other — a modularity pattern OAA demonstrated two and a half decades before the current agent boom.

The Trust Rule: The Verifier Must Not Grade Its Own Homework

Self-verification loops are an active research direction. Agents that mine their own failed attempts — retrospective self-refinement, failure-makes-the-agent-stronger reward loops — are valuable for assistants that can afford to retry. They are dangerous for verifiers. If the agent decides both what to check and whether the check passed, the audit trail collapses.

This concern has formal roots. Linearizability (Herlihy & Wing, 1990) defines correctness for concurrent objects by requiring each operation to appear to take effect atomically at a single point between its invocation and response. A verification agent that marks a test as pass or fail must linearize with the actual UI state. If the agent delays the decision or reinterprets ambiguous evidence, that single point drifts. The only safe separation is architectural: the perception function observes, the evaluation function judges.

The lane enforces this through a strict architectural gate. The evaluate_route function runs pure, browserless assertion logic: HTTP status code (owner-gated 307 redirects to sign-in treated as expected passes), horizontal overflow beyond 1 px, expected text content, required selector presence, and console errors after filtering a small benign allowlist (currently a single entry — the Vercel Insights script). The LLM triage is optional and env-gated (VERIFY_UI_LLM_TRIAGE=1). When active, it may append an explanation to a failure — it can never flip a verdict.

The self-test verifies this boundary programmatically. Running uv run python -m kg.verify_ui --selftest exercises the pure evaluate_route logic on canned inputs, then asserts that neither "playwright" nor "PIL" has entered sys.modules — the browser and image dependencies stay lazy. The boundary is enforced, not aspirational.

Build Tour: The Verification Core

The lane lives in roadmap-kg/kg/verify_ui.py, just over 400 lines. The driver function (verify_routes) launches headless Chromium via playwright-python, navigates to the built standalone server, waits for the load event (never networkidle — a polling page keeps the network busy forever), screenshots to .verify-ui/<slug>.png, and hands the observations to the pure assertion function (evaluate_route).

The built-server discipline eliminates the largest source of flakiness in modern web testing. Against the built server, every page is a static export; the load event fires once and the content does not change. Dev-mode hydration noise produces false verdicts — avoid it.

The lazy-import pattern is the architectural gate. The excerpt below is condensed from roadmap-kg/kg/verify_ui.py — the verdict function imports nothing from a browser, and the browser only ever appears inside the driver:

# excerpted from roadmap-kg/kg/verify_ui.py

# PURE verdict — no browser import anywhere in this function
def evaluate_route(initial_status, final_url, overflow_px, title, body_text,
                   console_errors, spec, *, selector_present=None) -> RouteResult:
    failures = []
    if initial_status >= 400:
        failures.append(f"HTTP {initial_status}")
    if overflow_px > 1:
        failures.append(f"horizontal overflow {overflow_px}px")
    # ... expected-text, required-selector, and console-error checks ...
    return RouteResult(route=spec.route, ok=not failures, failures=failures)

# the browser lives only inside the driver, behind a lazy import
def verify_routes(specs, *, base_url, **kw):
    from playwright.sync_api import sync_playwright   # LAZY — never at module top
    ...

# the selftest proves the boundary holds
def _selftest():
    import sys
    assert evaluate_route(200, "/x", 0.0, "OK", "body", [], clean,
                          selector_present=True).ok
    assert "playwright" not in sys.modules   # pure path never pulled the browser

The self-test runs with no browser installed and confirms the pure logic produces verdicts without any browser dependency, keeping the verification core testable in isolation.

Each route runs the same five deterministic checks — HTTP status (with owner-gated 307s treated as expected), horizontal overflow, expected text, required selector, and filtered console errors — plus an optional visual-diff comparison when a baseline directory is supplied. The last committed sweep (data/verify-ui-report.json, generated 2026-07-05T17:45Z) checked 10 routes and all 10 passed with 0 failures. The flakiness rate for deterministic assertions is zero by construction — there are no timing windows, no element-wait strategies, no network-idle races.

v2: Bounded Self-Navigation and the 64×64 Visual Diff

The v1 lane deferred the two features that make it look like the research papers: self-navigation and visual regression. v2 adds both with bounds that preserve determinism.

Discovery BFS (--discover --depth 2 --max-routes 25) takes a seed route, crawls same-origin links (stripping fragments and query strings, dropping /api/, /auth/, /_next/), sorts the routes deterministically, and runs the full assert suite on every discovered page. The BFS visits each route once; the --max-routes cap prevents uncontrolled explosion, and the deterministic sort makes the discovered set reproducible across runs. That turns a hand-maintained route list into a self-updating one — new pages that link off an existing seed get verified automatically, without adding a single test script.

Visual-diff baselines (--baseline / --update-baseline --diff-threshold) compare screenshots against a baseline directory. The lane downscales both images to 64×64 grayscale and computes the mean absolute pixel difference. Default threshold 2%. Lazy-imported Pillow handles the processing; if missing, the lane emits a SKIP note and proceeds with deterministic asserts.

At 64×64, the diff catches layout collapse, missing chrome, and large misalignments — but not one-pixel regressions. That is a stated trade-off against flakiness. PNG encoder differences that make byte-level compares unstable vanish at this resolution, so the default 2% threshold stays quiet on encoder-level noise while still flagging structural change. (The visual-diff lane is opt-in via --baseline; the committed sweep runs the deterministic asserts without it.)

The Closed Loop: The Page That Verifies Itself

The lane outputs a machine-readable report (--out JSON) committed as data/verify-ui-report.json. A /computer-use page renders this report. The sweep’s route list includes that very page.

Every run verifies the page that publishes its verification results. If the report page renders incorrectly — wrong data, broken chart, missing route list — the disparity is caught in the next sweep. The system becomes self-referential: its verification includes the condition of its own reporting infrastructure.

This property is rare in test suites and worth borrowing. Ensure that your verification pipeline tests the surface that reports its status. The committed report from 2026-07-05 carries generatedAt: "2026-07-05T17:45:32+00:00", total: 10, failed: 0, with a per-route results array. Re-run make verify-ui before publishing and the /computer-use page picks up the fresh numbers.

How This Fits the Research Landscape: None of the Papers Address Broken UIs

The foundational papers provide essential components, but every one assumes a clean, working UI. Here is how my system fills that gap:

Rico (Deka et al., UIST 2017) – more than 72,000 unique UI screens mined from over 9,700 Android apps, the largest mobile UI repository, and a common substrate for training UI state-recognition models. Critically, none of those screenshots contain deliberate errors — no missing labels, no overlapping elements, no console errors. A verification agent trained purely on Rico would be blind to real-world regressions. My lane compensates by checking runtime conditions (console errors, overflow) on the live built UI, not on clean snapshots.

PIXELHELP (Li et al., ACL 2020) – the corpus pairs natural-language instructions with the mobile-UI action sequences that carry them out; the paper's best model reaches 70.59% accuracy on predicting complete ground-truth sequences. The gap between instruction and action means agents must make non-obvious decisions — exactly which element to tap, how long to wait. My lane sidesteps this entirely: routes are URLs, assertions are explicit. Less flexible, far more reliable.

Open Agent Architecture (Martin, Cheyer & Moran, 1999) – distributed agents brokered by a facilitator that matches requests against advertised capabilities. While not UI-specific, the modularity principle applies. My lane has three internal components (navigator, verifier, reporter), each swap-able. A Rico-trained vision model could replace the visual diff; a PIXELHELP-style parser could replace the BFS route discovery.

Linearizability (Herlihy & Wing, 1990) and Algebraic Process Verification (Groote & Reniers, 2001) provide formal frameworks that my lane does not fully implement. The deterministic assert layer gives a linearizability-style guarantee per check: the verdict reflects the UI state at screenshot time. But proving that the agent’s entire tap-scroll-tap chain satisfies a temporal specification — e.g., "after login, dashboard must load within 2 seconds" — remains open.

OSWorld 2.0 (arXiv:2606.29537) resets expectations: best binary completion 20.6%, partial 54.8% for Claude Opus 4.8. My lane’s deterministic asserts pass every known route when the UI is correct (10/10 in the last committed run). The gap between 20.6% and a clean sweep defines the difference between an agent operating unfamiliar UIs and one operating its own known surface.

Volume trend — an OpenAlex phrase query returns 32 (2023) → 268 (2024) → 743 (2025) → 2,003 so far in 2026. Annualized ~4,000, a 5× year-over-year pace. (Caveat: the "browser agent" phrase catches some non-agent UI testing work, so read this as directional, not a census.)

None of the foundational papers handle adversarial or broken UIs. Rico’s clean screenshots, PIXELHELP’s clean demonstrations, OAA’s cooperative agents — all assume the UI is working. Verification agents must operate on the opposite: pages with hydration errors, missing assets, misconfigured redirects. The lane addresses this by checking real runtime conditions, but the research community has not yet produced datasets or models specifically for UI verification on broken UIs.

Decision Framework: When to Use Which Approach

Criterion	Deterministic Verification Agent	Traditional DOM-Based Tests	LLM-Powered Testing Agent
Reliability	High (100% pass on correct UI, 0% flaky on 10-route run)	High (locator-based, but brittle to DOM changes)	Low (OSWorld 2.0: 20.6% completion on complex tasks)
Maintenance cost	Low (no element selectors to update)	Medium-high (selectors break on refactors)	Medium (prompts may need updating)
Coverage of visual issues	Medium (64×64 coarse diff, overflow, console – catches layout collapse)	Low (no visual checks unless screenshot comparison added)	High (vision models can detect layout issues, but unreliable)
Speed	Fast (headless nav + pure asserts, no model inference)	Fastest (DOM query directly)	Slow (LLM inference latency, often seconds per check)
Cost per run	Near zero (no LLM tokens for verification)	Near zero	High (model inference costs)
Flakiness rate	0% (deterministic asserts, no timing windows)	Low-moderate (locator timing, network delays)	High (model variance, non-deterministic output)
Ability to adapt to UI changes	None (must re-run with new routes or BFS)	Low (broken selectors need fixing)	High (vision models adapt, but accuracy drops)
Suitability for CI	Excellent (deterministic, fast, no flaky LLM)	Good (but flaky locators cause intermittent failures)	Poor (20% pass rate unacceptable for CI gating)

The lane occupies a specific niche: verifying a known set of critical pages on every deploy. It is not suited for exploratory testing, accessibility auditing, or load testing. For "does the page load without console errors" and "are all critical elements visible," deterministic asserts win.

Challenges and Open Problems

The lane is honest about its limitations. The BFS mode adds agency, but the assert logic remains scripted. The 64×64 visual diff is coarse — it catches layout collapse but misses a missing icon or a 2px alignment shift. Pixel-perfect lanes exist for teams that need them, at the cost of higher flakiness.

The benchmark numbers are moving targets. The OSWorld 2.0 figure of 20.6% is from a preprint about a brand-new benchmark — expect it to be outdated within months. The OpenAlex volume trend of ~4,000 annualized papers is a signal, not a precise census. Treat all numbers here as directional, current as of their retrieval date (see citations).

The foundational papers from this brief — Rico, PIXELHELP, OAA, linearizability, process verification — are all pre-2026. The field is moving fast, and none of these works address the core verification gap: operating on broken UIs. My lane compensates by checking runtime conditions, but the research community needs datasets and benchmarks specifically for UI verification on failure states.

The Unglamorous Half of Computer Use

Computer-use agents capable of operating any UI are not yet reliable enough for unsupervised verification. The OSWorld 2.0 reset makes that clear: even with max thinking, the best model fails four-fifths of the time.

But a bounded verification agent that operates only your own UI, with deterministic assertions as the decider and an LLM as an optional explainer, works today. It costs near zero to run, catches real regressions, and closes the loop by verifying the page that publishes its results. The boundary that matters is not between scripted tests and agents — it is between a system that grades its own homework and one that doesn’t.

The next step is integrating formal verification of action sequences — proving that the agent’s tap-scroll-tap chain satisfies a temporal specification, as Groote and Reniers (2001) outlined for processes. Until then, deterministic asserts with bounded exploration are the most trustworthy pattern for UI verification. The agent explores; the code judges. And the page that displays the verdicts is itself under verification.

---

The OSWorld 2.0 figures are from arXiv:2606.29537 (submitted 2026-06-28). The volume trend is from an OpenAlex phrase query re-run 2026-07-06 (directional signal, not a census). The build details and the 10-route sweep are from my own codebase — roadmap-kg/kg/verify_ui.py and the committed data/verify-ui-report.json. The skill-distillation references are COLLEAGUE.SKILL (arXiv:2605.31264) and online skill learning for web agents (arXiv:2606.04391).