Autonomous trading: the board's biggest ratio misses by 0.003
The largest spread-to-standard-error ratio on the board is 4.2996, and it is not significant. It has to clear 4.302653 — the two-sided 5% critical value on 2 degrees of freedom. That is not the 1.959964 the rest of the one-day table is measured against. 4.2996 falls 0.003 short. Nothing about the number is wrong; it was judged against a bar that belonged to a different sample size — the error Bailey and López de Prado (2014) built the deflated Sharpe ratio to catch, where the length of the track record behind a statistic is part of the threshold and not a footnote to it.
That gap is worth a long article not because 0.003 is large, but because the board prints no column that says so, and because the system that produced the number declined to promote, demote or score anything on the strength of it. The failure mode has a name. Bailey and López de Prado (2014) describe it as an undeflated ratio: a performance statistic reported without controlling for the number of trials behind it, the length of the track record, and the non-normality of the sample. The correction applied to this cell is the crudest possible version of their adjustment — the one that comes free with a t-table.
A three-period lane can outrank a 981-period lane without anyone lying about the arithmetic
The screen ranks several thousand names cross-sectionally. One lane ranks them by how often they are mentioned on public social channels, a raw attention count; a companion lane ranks them by the change in that count. Neither could be scored until the panel held at least one ranking date with a priced next session on the other side of it — and overnight that countdown ran out. The feeder pulled the previous session's grouped closes, re-sealed the panel over 982 dates, and both lanes scored at the one-day horizon for the first time: the number of scorable periods moved from 2 to 3. The five-day horizon sits at zero periods, countdown at one.
The mention-count lane, top 25 names, one-day horizon: +102.89 bps net spread, standard error 23.93 bps, on 3 periods. The mention-change lane, same cut: -15.60 bps at a standard error of 18.68 bps, with per-date signs of [-, -, +].
The spread convention matters more than the number, and the record quotes it verbatim, which is the right instinct. For ranking date d, the one-day forward return of ticker t is (close[d+1] - close[d]) / close[d], NaN where either end is missing. The lane's spread is the equal-weighted mean over its top-25 names minus the equal-weighted mean over the whole screened set on d. The reported net figure is the mean of the per-date spreads, and the standard error is the sample standard deviation over the square root of the period count.
Periods are counted as periods, never as name-periods — 3 periods here is 75 name-periods, and the record says 3. Zhang et al. (2026) built an entire benchmark around the principle that a return whose execution convention is not stated cannot be checked for protocol-induced inflation. The same discipline applies to counting units. Seventy-five correlated name-periods inside a single session is not seventy-five independent draws, and a board that quietly swapped between the two units would be unusable in exactly the situation where it matters most.
The concentration test that was supposed to kill it, and the replication that agreed to 0.0 bps
The first thing to try to kill a three-day result with is concentration: one date wearing a mean's clothes. It does not hold here.
All three per-date spreads on the mention-count lane are positive — +112.4396 bps, +138.7427 bps, +57.4984 bps — and the mean without the largest of them is +84.97 bps. Three positive draws in a row is also the shape Daniel, Hirshleifer and Subrahmanyam (1998) derive from biased self-attribution: overconfidence about the precision of private information, combined with asymmetric confidence shifts after outcomes, produces positive short-lag autocorrelation — the "momentum" they pair with negative long-lag autocorrelation. Their theory says three consecutive positive days is exactly what a short-horizon momentum story looks like. That is a reason to look harder, not a reason to relax.
Then the replication. An independent leg took the ranked ticker identities from the backfill and every closing price from the raw vendor grouped files, calling none of the panel loader, the forward-return helper or the evaluator. Both lanes' net spreads and standard errors agree with the module's own report to 0.0 bps — an exact match, not a tolerance. That is a genuinely useful check, and it is not the check people will read it as.
Qian et al. (2025) open the Agent Market Arena paper by naming "rely on unverified data" as one of the three reasons it remains unclear whether trading agents can reason in live markets; an independent re-derivation from raw vendor files answers that complaint for this cell. What it does not do is produce a second sample path. It re-computes the same three sessions from the same underlying prices, so it tests the code, not the signal. Agreement at 0.0 bps tells you the pipeline is not lying to you, and nothing about whether three days is enough.
The ratio looked overwhelming against the wrong bar, and the correct bar was hiding in the period count
This is the reversal, and it is the whole point of the piece.
The ratio of net spread to standard error on the mention-count lane is 4.2996 — 102.89 bps divided by 23.93 bps — and that is the largest such ratio anywhere on the board. The other one-day cells carry period counts of 276, 527 and 981, and at those sizes the ratio is a roughly Gaussian object measured against a 1.959964 bar. Against 1.959964, 4.2996 is not near the line; it is over it by a factor of roughly 2.2. Shown only the ratio and the bar, I would have called it the strongest thing on the table. That instinct is the one Bailey, Borwein, López de Prado and Zhu (2013) built a method against: when a result is selected as the best of many simulated configurations, standard hold-out reasoning becomes unreliable, and the quantity that matters is the probability the selection itself is overfit. 4.2996 is not one test. It is the maximum over every cell on the board.
That judgement is wrong, and it is wrong in a way worth internalising. 4.2996 rests on 3 periods, which is 2 degrees of freedom, and the two-sided 5% critical value on 2 degrees of freedom is 4.302653. The largest-looking ratio on the board misses the bar its own sample size implies by 0.003. Nothing about the ratio changed; the bar did.
This is precisely the deflation Bailey and López de Prado (2014) argue for, applied at the crudest available resolution: significance is a property of the test, not of the headline statistic, and a t-table encodes the sample length for free.
The record's posture after noticing this is as instructive as the arithmetic. No threshold in any module was read, moved or proposed. No verdict was written. The lane was not promoted, demoted or scored. The critical value is filed as the arithmetic companion of a period count, not as a test that anything passed or failed, and the gap became an audit item. That restraint does real work, because the temptation at this point runs the other way: the number looks good, so the bar starts to look like an obstacle.
The board prints a fold-count warning and nothing at all for the column that actually failed
The screen already knows how to flag small samples. It prints a warning that a one-date lane's 1-of-1 outranks a 235-date lane's 4-of-5 — a warning about fold counts, where the same degrees-of-freedom problem appears in a different costume. The standard-error column has no such line. A reader scanning the board sees 4.2996 sitting next to 1.959964 and has to know, from outside the table, that the bar for that row is somewhere else entirely.
Yehudai et al. (2025) map the agent-evaluation literature across four dimensions — fundamental capabilities, application-specific benchmarks, generalist benchmarks, and evaluation frameworks — and the complaint running through their survey has this shape: the gap is rarely the metric itself, it is what the metric fails to disclose about its own conditions.
The second place the table misleads is the one that looked like a defect and was not. These two cells print wide-interval standard errors of 15.46 and 9.41, both below the IID standard error, and a reader who knows only the textbook expectation would read "below IID" as a bug. It is not. The estimator clamps its lag to n-1 and returns nothing on a non-positive long-run variance.
At 3 points with negative first-order autocorrelation, that is a documented, guarded estimator on a very short series, not an unguarded one. Paskaramoorthy, van Zyl and Gebbie (2025) show that the bias introduced by mishandling the return autocovariance structure in backtests is bounded by, and strongly dependent on, the observable first-lag autocorrelation — the same quantity the estimator is defending against when it refuses to compute. A guard that returns nothing, or a wide interval, on three points is the behaviour you want. The failure is presentational: the column does not say why.
A missed tick cost 25% of this lane's history, and the completeness gate is not the lane's clock
The lane recorded its first missing date, and the miss is not a join failure. For that session no capture file exists at all. A tick did not run. That date is 1 of the 4 sessions the lane was eligible for, so a single missed capture cost 25% of this lane's entire scorable history, and no vendor endpoint can return it, because the aggregator serves no history. This is the unglamorous half of what Qian et al. (2025) build into Agent Market Arena when they insist on verified trading data and continuous, real-time capture as a precondition for comparing agents at all: a live evaluation is only ever as long as its least reliable collector. A missed cron job is not a statistical problem and no statistical correction touches it.
Fang et al. (2025) frame self-evolving agents as a feedback loop over System Inputs, Agent System, Environment and Optimisers, and their central complaint is that most agent systems rely on configurations that remain static after deployment. Here the loop ran, the feedback closed, and the missing input was not something optimisation could recover. Data that was never captured is not a low-priority backlog item; it is a permanently absent observation, priced at a quarter of the lane's history.
There is a second arithmetic fact about the capture schedule that has nothing to do with the tick that failed. Of the 8 captures on disk, 3 name non-trading days — two weekend dates and a market holiday — and those can never join to a forward return. Capturing daily does not make the period count grow daily.
Mohammadi et al. (2025) organise agent evaluation along objectives and process, and single out dynamic, long-horizon interaction as an enterprise-specific challenge the literature routinely overlooks. A capture scheduler that writes files on weekends is a small version of that oversight: the process metric improves while the evaluation surface does not.
Then there is the completeness figure, which reads 0.0031 for this lane and is tempting to read as a countdown. It is not, because its denominator is every scorable date, and that denominator grows by one per session alongside the numerator. Solving (3 + N) / (982 + N) ≥ 0.95 gives N = 18,598 sessions — 73.8 trading years. That is a consequence of the gate's own definition and emphatically not an argument to move it; the gate stays where it is.
Qian et al. (2025) list "cover limited periods and assets" as one of the three defects that make live trading agents hard to assess, and their answer is a lifelong arena rather than a cleverer short-window metric. The clock that actually moves here is the period count, at one period per trading session: 57 sessions to 60 periods, 249 sessions to 252. The honest re-read dates are already known.
The literature's prior for a top-25-by-attention basket is negative drift, which makes three positive days the wrong thing to celebrate
The standing prior for a basket selected by social mention count is not positive. Han and Kumar (2013) show that stocks with a high retail trading proportion have strong lottery features, attract gamblers, tend to be overpriced, and earn significantly negative alpha: the average monthly return differential between the extreme retail-proportion quintiles is -0.60%. Names that dominate a public mention count are close cousins of that population, so the expected sign of a top-25-by-attention spread is negative — and if you compound -0.60% monthly, that is roughly 7% a year of negative drift riding on a basket like that. A positive three-day spread of +102.89 bps is therefore the surprising direction.
There is the stake, stated plainly: promote this lane on a 1.96 bar, size it, and the population you are buying is the one Han and Kumar (2013) measure as overpriced and underperforming — stocks with a high retail trading proportion carry strong lottery features, attract gamblers, and earn significantly negative alpha, with a monthly return differential between the extreme quintiles of -0.60%. A wonder measured on three days is the single most likely place for a sample-size artefact to hide.
The other half of the prior is about shape rather than sign. Daniel, Hirshleifer and Subrahmanyam (1998) derive positive short-lag autocorrelation, short-run earnings drift, excess volatility, and negative long-lag autocorrelation from overconfidence and biased self-attribution — plus a negative correlation between future returns and long-term past performance. A lane that shows three positive days and then decays toward the other lanes' magnitudes as the sample grows is the shape their model predicts.
That is not evidence against the lane. It is a description of what an honest reading of a three-day sample looks like, and it makes "re-read at 10, at 20, at 60" the correct next action rather than a compromise position.
The benchmark literature's answer to a short window is a longer window under a fixed protocol
Xia et al. (2026) audited 77 studies of LLM trading agents in a protocol-coded snapshot screened through 2026-03-09 and found that only 19 satisfy their minimum boundary of Action Output plus Closed-Loop Evaluation, with the remaining 58 retained as design context. Inside the primary subset the numbers are brutal: 2/19 report extractable time-consistent split protocols, 1/19 reports an explicit transaction-cost model, 1/19 documents universe or survivorship handling, 11/19 report execution timing or semantics, 15/19 are coded R0, and none reach R3 reproducibility. Their conclusion is not that the field is wrong about trading; it is that protocol incomparability makes its results unrankable.
Read the record above against Xia et al.'s checklist and it does surprisingly well — the spread convention is quoted verbatim, the comparison set is defined as the whole screened set, the cost treatment is stated as net, the period count is stated rather than implied. What it fails is the last column: 3 scorable periods against a 982-date panel, and no amount of convention discipline fixes that.
Zhang et al. (2026) give the sharper version of the same lesson. Their one-switch benchmark toggles one evaluation convention at a time around a clean t+1-open reference while holding the data panel, the walk-forward split, the model family, the horizon, the portfolio rule and the cost convention fixed, across two daily-OHLCV equity panels, six model families and yearly tests from 2016 to 2024. Inflation turns out to be highly selective: same-day-open execution with post-open daily-bar information inflates badly and stably, while same-day-close execution is weak in most settings. They also state plainly that the benchmark is diagnostic rather than a claim of tradable alpha — which is the correct status for +102.89 bps as well, and why the spread convention had to be quoted verbatim before anything else in this discussion could be trusted.
Fan et al. (2025) close the loop on what a promising number does not tell you. AI-Trader is a live, data-uncontaminated benchmark spanning three markets — US stocks, A-shares and cryptocurrencies — evaluated across six mainstream LLMs and multiple trading frequencies, with agents given only minimal context and required to search, verify and synthesise live information themselves. Their striking finding is that general intelligence does not automatically translate into effective trading capability: most agents post poor returns and weak risk management, and risk control capability, not model strength, determines cross-market robustness.
A lane that tops a board is a model-side result. Whether it survives execution, turnover and capacity is a separate question that three days of price data cannot answer.
Practical takeaways: a framework for ratios you cannot yet deflate
Derive the bar from the period count and print it on the row it applies to. The board's other one-day cells can be judged against 1.959964 because they carry 276, 527 and 981 periods. This cell cannot, because it carries 3 and therefore 2 degrees of freedom. A table that prints one bar for every row is printing a number that is correct for most rows and wrong for the rows that matter most — and the wrongness is invisible precisely where the numbers look best.
State the number of trials. The ratio 4.2996 is not one test; it is the maximum over every cell on the board. Bailey and López de Prado (2014) exist in the literature because the maximum of many undeflated ratios is an optimistic object by construction. When you find the largest number on a board, the first question is how many numbers you had to look at to find it.
Separate the clock you can wait on from the clock you cannot. The completeness gate reads 0.0031 and implies 73.8 trading years of waiting at the 0.95 level; the period count moves at one per trading session, putting 60 periods 57 sessions away. Only one of those is actionable, and the record held both numbers without moving the gate.
Fix the data supply before arguing about the statistics. A missed tick cost 25% of this lane's scorable history and the aggregator serves no history, so the capture file is the only copy that will ever exist. Of the 8 captures on disk, 3 name non-trading days. Neither of those is a statistical problem, and no statistical correction touches either.
Pre-commit the re-read. The cheapest honest answer to a three-period result is a schedule: re-read the cell at 10 periods, at 20, and at 60, and ask whether the ratio is still the largest on the board. If a lane decays toward the other lanes' magnitudes as the sample grows, that is the prediction a three-day sample makes, and it costs nothing to test. Decay is also the base rate: McLean and Pontiff (2012) tracked 56 published cross-sectional return predictors and found an average out-of-sample decay of 15% and an average post-publication decay of about 50%. Committing to the re-read schedule before the numbers arrive is what keeps that decay from being renegotiated once it does.
The interesting event was an optimiser that declined to promote anything
The board produced its largest-ever ratio. The loop added a period, recomputed the cell, noticed that the cell's own degrees of freedom disqualified the result by 0.003, wrote that gap down as a defect in its own reporting surface, and changed no threshold. Fang et al. (2025) observe that the self-evolution literature is thick with proposals for agents that improve themselves and thin on systems that actually adapt after deployment. An agent declining to act on a favourable number is a rarer artefact than the number itself.
+102.89 bps on 3 periods is a measurement of three days. It does not say the lane is a signal. It does not say the lane is not a signal. It says nothing about the five-day horizon, which is still at zero periods, and it proposes no bar.
A reading in either direction is a conclusion nobody is entitled to draw from this record, and the interesting question the loop left itself is a waiting question rather than an arguing one: does the +102.89 persist as the period count grows, and at what point does it stop being the board's largest ratio? Fang et al. (2025) frame self-evolving systems as a feedback loop over inputs, agent, environment and optimiser; an optimiser whose output on its own best number is "wait 57 sessions" is operating that loop correctly, not failing to.
The answer is 57 sessions away. In the meantime, the work that is actually available is the column: something on the board that tells the next reader that a standard error resting on 2 degrees of freedom is judged against 4.302653 and not against 1.959964. That column is worth more than the ratio that made it necessary. Yehudai et al. (2025) end their survey of agent evaluation on the same note: the gaps that matter are in what an evaluation fails to disclose about its own conditions, not in the headline it reports.