An optimistic backtest can show Sharpe 5 where live reality is negative, and an AI-first crypto trading system graded by it will optimise into the gap with total conviction — because inside that simulator the strategy genuinely works.

The simulator is not a test. It is the model's reward function. That is the sharpest trap I know, and the rest of this post is what I found while walking into it.

Here is the shape of what I found. Give a router a third action — quote, cross, or abstain — and it takes the third. Doing nothing scores zero; every alternative scores less; and zero wins 11 of 11 panels without the signal being consulted at all. The fitted policies that do trade pick about 1.6% of rows and still end below zero. A model that has learned to almost-not-play is not broken. It is reporting the absence of an edge — the one output no trade-count metric will ever reward.

Everything that decides whether such a system makes money on a perp lives in the coupling between model and market: fees, funding, regime, and the evidence you are willing to accept. A round trip costs 4–14 bps before the model says a word.

I went looking for that edge at sub-minute horizons and did not find it. What the search produced instead was thirty-seven principles about how to run a model against a market — each ending with the condition that breaks it, each carrying the measured number behind it, and several carrying the number that killed an earlier version of the same claim. They are worth more than the strategy would have been.

Most write-ups about concurrency in LlamaIndex tell you half the story. They show you how to decorate a step with num_workers=5 and call it a day. What they don't tell you is that the storage layer under those parallel steps fails in opposite ways. The default 4 workers guarantee you'll hit both failure modes eventually. I’ve timed the workflow side on llama-index-workflows 2.22.2 (PyPI release) and I’ve lived through the silent corpus inflation and lock-contention meltdowns that happen when those workers hammer the vector store. Here’s the full picture.

Most AI agents you deploy today are frozen the moment they go live. You handcraft the prompts, select the tools, wire up the memory, and hope the configuration survives contact with real users. It doesn't. Tasks drift, APIs change, user intents shift – and your agent silently degrades. The conventional fix is another round of manual reconfiguration. But there's a more principled path: treat agent design not as a one-time assembly but as a continuous search problem.

Most self-evolving agent demonstrations—those that appear to learn by picking better tools or adjusting dialogue style—avoid modifying the core reasoning engine. Evolving the reasoner itself—the chain-of-thought architecture, the internal planning logic, the very way an agent thinks—is the hard, brittle, data-starved problem that separates parlor tricks from genuine lifelong adaptation.

Most teams building self-evolving agents obsess over prompt engineering or fine-tuning the LLM. They miss the bigger lever: the substrate—what the agent remembers and which tools it wields. A prompt is ephemeral; memory and tools are structural. Evolving the substrate yields compounding returns that no amount of prompt tweaking can match. Fang et al. (2025) survey of self-evolving agents confirms this: the components that persist across sessions—memory and tools—define the agent's operational range far more than any instruction string. In this third part of the series, I'll lay out why memory and tool optimisation are the neglected backbone of lifelong agent systems, back every claim with data from the literature, and give you a decision framework you can implement today.

Here’s the uncomfortable truth the hype cycle doesn’t want you to hear: a single, well-prompted model often beats an entire team of specialised agents on standard reasoning benchmarks. Pan et al. (2025a) demonstrated that single large LLMs with carefully crafted prompts can match the performance of complex multi-agent discussion frameworks across multiple reasoning tasks arxiv:2508.07407. Jwalapuram et al. (2026) push the finding further: a single-agent GPT-5 instance using chain-of-thought with self-consistency “reliably outperforms the most sophisticated GPT-4o-based MAS frameworks (e.g., ADAS or AFlow) while consuming less than half the total tokens,” and automatically generated multi-agent systems “consistently underperform CoT-SC despite being up to 10x more expensive” arxiv:2606.13003. If you’re building an agent system and your first instinct is “let’s spin up three agents and make them debate,” you might just be burning tokens for no gain.

The moment an agent can rewrite its own code, evaluation ceases to measure and starts to train.

Two agents answer the same user query. Both return the identical string—correct, well-formatted, cited. An answer-level eval gives them both a perfect score, identical down to the decimal.

One agent made three redundant retrieval calls (same tool, same query, same corpus) before stumbling on the right source. The other called exactly the right tool once and answered. The answer-level eval cannot tell the difference. It never could.

The keys are in the trajectory.

I built a trajectory observability lane for my agents in three small pieces: the JSONL traces every workflow already emits but nobody reads, a judge that scores the tool-call sequence instead of the answer, and a Langfuse uploader written against the raw REST ingestion API—no SDK. Publication volumes indicate this is the moment: agent-observability research jumped sharply into 2026 (the phrase barely existed before), and the first dedicated fault-detection benchmark for agent observability was published this week.

This post is the full walkthrough: what trajectory observability is, why answer-level evals miss half the story, the three-module build, and how the research on partial observability validates the approach.

In late June 2026, the best computer-use agent failed four of five tasks on the new OSWorld 2.0 benchmark. Claude Opus 4.8 with max thinking hit just 20.6% binary completion on 108 long-horizon tasks, each requiring a median 1.6 hours of human interaction and roughly 318 tool calls. GPT-5.5 plateaued near 13%. (OSWorld 2.0, arXiv:2606.29537, submitted 2026-06-28.)

That reset changed what I think verification agents should be. Most researchers saw a challenge: build better agents that operate other people’s software. I saw the inversion. A computer-use agent that wanders any UI is too unreliable for unsupervised verification. But a bounded agent that operates only your own UI, with deterministic assertions as the decider and an LLM as an optional explainer, works today. It costs near zero per run, catches real regressions, and never grades its own homework.

This post documents the lane I built for my own site: just over 400 lines of Playwright-driven verification, a BFS crawler for route discovery, a 64×64 visual diff, and a closed loop where the page publishing the results is itself under verification. I’ll show how it draws from foundational research — OSWorld 2.0, the Rico, PIXELHELP, the Open Agent Architecture — and where those papers fall short for verification on broken UIs.

Durable execution for LLM agents ensures that an agent workflow can survive infrastructure failures by persisting its state at each step, allowing automatic retry and resumption without data loss or duplicate side effects — using patterns like idempotency keys, event sourcing, and workflow engines such as Temporal.

Picture an agent mid-run when the server dies—someone hits Ctrl-C on the wrong terminal, a redeploy rolls the process, the host reclaims the instance. In a naive setup, that run is simply gone. In a durable one, it isn’t: GET /workflows/runs/ABC123 returns status: waiting_human with the full conversation history intact. A moment earlier that workflow was a fragile in-memory object; kill the process and it would have vanished. The difference is three small modules I built into my own FastAPI workflow service: a SQLite store, a runner that snapshots the context after every event, and a route to resume parked runs. Durable execution for LLM agents, at the simplest scale that still works, looks like this.

The design fork every builder hits is the real decision: checkpoint the state, or journal the events. Underneath it sits a silent assumption—that long-running agents can restart after a crash—that the academic literature has only just begun to examine (Ding et al., 2026).