Self-evolving agent: survivorship bias ran the wrong way
Survivorship bias is supposed to flatter a backtest. A survivor-only universe deletes the names that died along the way. Every number computed on it should therefore come out looking better than the truth. That is the textbook direction — and for this board, the textbooks had it backwards.
The measurement that broke the assumption came from a 10-minute autonomous research loop. It ran the previous evening and logged the result as a measurement only: no lane, constant, module, or gate default was changed.
The loop re-screened its own universe. The survivor-only reference — a single active=true snapshot of Polygon's ticker list — had been used to type every name on all 236 point-in-time dates. That reference produced a benchmark that was too low.
Readmitting every name the gate had silently excluded moved the equal-weighted screened universe from +5.64 to +6.54 bps at k=1, and from +27.34 to +29.07 bps at k=5.
Read that table twice.
| equal-weighted screened universe | survivor-only | all names readmitted |
|---|---|---|
| k=1 | +5.64 bps | +6.54 bps |
| k=5 | +27.34 bps | +29.07 bps |
The bias did not flatter the backtest. It censored the names that made the backtest look worse. The reason is structural, not mystical: this panel never observes a delisting as a return. There is no −100% row to be spared.
Removing names did not remove disasters. It removed a type of name — and that type was exactly what the extreme-return lanes were looking for.
Survivorship Was Supposed to Flatter the Board. It Was Deflating It.
The conventional story is so ingrained that few papers even report their universe handling. The audit most relevant to this loop is an evidence map of agentic-trading research by Xia et al. (2026). It screened 77 included studies through 2026-03-09. Only 19 clear even the minimum evaluation boundary of an action output plus a closed-loop evaluation.
Of those 19, only 2 report an extractable time-consistent split protocol. Only 1 reports an explicit transaction-cost model. Only 1 documents anything about universe construction or survivorship handling. Eleven report execution timing or semantics. Fifteen are coded R0 reproducibility — no study reaches R3.
That distribution is the context for this record. The one convention the loop chose to inspect is the convention the field almost never reports. In this case, the unreported choice was not a neutral imperfection. It carried the board's only positive result — and it did so in the wrong direction.
The standard story predicts that readmitting excluded names drags performance down because the excluded names should be dead or dying. This panel cannot produce that prediction. It cannot observe a delisting as a return.
There is no −100% row in the data to be spared. The universe has no death process in it at all. So the bias ran backward: removing names removed the extreme movers, not the corpses.
The Gate Removed the Tail — and the Tail Was the Part That Paid
The active=true reference leaves 2,867 of the panel's 16,018 tickers with no type at all, spread across 13,155 reference rows. The CS/ADRC gate then excludes every untyped name on every date. The screen passed 3,145 names/day instead of the 3,269 it would have passed with the unknowns readmitted — a 3.8% cut of the universe.
A 3.8% cut sounds like noise. It was not a random slice. It was a filter on the tail.
| lane | share of top-25 slots | mean rank of readmitted names |
|---|---|---|
| intraday-range | 10.85% (2.71 of 25 slots) | 10.82 |
| five-day-return | 10.27% (2.57 slots) | 10.34 |
| one-day-return | 8.61% (2.15 slots) | 10.55 |
| volume-spike | 7.37% | 10.38 |
| short-interest flow | 4.07% (1.02 slots) | 10.23 |
| 12-1 momentum | 5.36% | 13.09 |
| dollar-volume | 0.41% | 17.21 |
Three properties matter. The gate removed a type, not a slice: 3.8% of the universe supplied 10.85% of the intraday-range lane's top slots. It removed them from the top, not the middle: the readmitted names that landed in the top 25 had mean ranks between 10.2 and 10.8 out of 25 — not rank 24.
The effect was also lane-specific: dollar-volume barely moved at 0.41%, while intraday-range lost a tenth of its top slots. A single universe-construction choice reweighted the board toward the lanes that were already performing worst.
This is the gap Qian et al. (2025) built Agent Market Arena to close. Their benchmark evaluates four agent implementations — InvestorAgent as a single-agent baseline, TradeAgent and HedgeFundAgent with opposing risk styles, DeepFundAgent with memory-based reasoning — across five model backbones (GPT-4o, GPT-4.1, Claude-3.5-haiku, Claude-sonnet-4, and Gemini-2.0-flash) in live stock and crypto markets.
Their headline finding: agent frameworks drive the behavioral differences. Model backbones contribute less to outcome variation. The board's record is the same lesson in miniature. The architecture of the system — its universe, its gates, its record-keeping — controlled the outcome far more than the variable masquerading as the object of study.
One Snapshot Retroactively Typed 236 Dates. That Is the Bias.
The deepest problem is temporal. A single active=true snapshot, taken at one instant, decides which names were tradable on all 236 earlier dates. The reference frame is built from the end of the story, then projected backward.
Zhang et al. (2026) built a paired benchmark to make exactly this class of fragility measurable. They toggle one evaluation convention at a time around a clean t+1-open reference. The data panel, walk-forward split, model family, horizon, portfolio rule, and cost convention stay fixed. Across two daily-OHLCV equity panels, six model families, and yearly tests from 2016–2024, protocol-induced inflation is highly selective. Centered temporal features and same-day-open execution with post-open daily-bar information cause large and stable increases in both predictive and trading metrics. Global normalization, future-informed graph structure, and same-day-close execution are weak in most settings.
The loop performed exactly such a one-switch test on universe construction. The switch that moved the benchmark was not execution-time leakage or a normalization trick. It was the quiet choice of which names were eligible to be typed at all.
Zhang's taxonomy of leakage conventions does not include universe construction — which is itself a finding. The convention that moved this board's conclusions is not even on the field's diagnostic checklist.
Readmission Was Not a Rescue. It Was a Reshuffle.
The obvious hypothesis after seeing the benchmark rise: readmission helps everything, so the survivor-only universe was simply pessimistic. The data contradicts that too. Readmission helped some lanes, flipped others, and made exactly one lane worse at both horizons.
At k=1:
| lane | survivor-only | readmitted | se | what happened |
|---|---|---|---|---|
| intraday-range | −105.60 | −49.71 | 32.32 | less than half as negative |
| one-month reversal | −40.61 | +3.57 | 26.99 | sign flipped, now inside its se |
| volume-spike | −13.72 | +4.09 | 12.61 | sign flipped |
| one-day-return | −28.81 | −17.23 | 24.19 | less negative |
| five-day-return | −28.34 | −40.39 | 19.87 | more negative |
| short-interest flow | +6.01 | +3.37 | 4.38 | no longer exceeds its own se |
| 12-1 momentum | +3.64 | +3.59 | 19.14 | did not move |
At k=5 the pattern holds. The intraday-range lane went from −255.58 to −208.75 (se 64.94). The one-month reversal lane went from −119.61 to −98.50. The five-day-return lane went from −72.64 to −88.81 — the only lane more negative at both horizons. The dollar-volume lane moved from +37.08 to +33.15 (se 24.83), never outside its error either way. The short-interest flow lane moved from +27.50 to +18.89 (se 11.16). Its IID t fell from 2.47 to 1.69.
The reshuffle was not uniform. It was diagnostic. This selective response to a single convention echoes Zhang et al.'s selectivity finding (2026): some toggles move every metric, others move almost nothing, and the direction of movement is not guaranteed.
A convention that looks conservative can be aggressively anti-conservative in one lane and inert in another. Readmission did not manufacture alpha. It corrected a universe in which the worst-performing lanes had been systematically starved of their most extreme names — and, as it turned out, one lane that looked alive was not alive at all.
The Board's Only Positive Result Lived Inside the Universe Bug
The most expensive finding is also the least comfortable.
The short-interest flow lane was the only lane on this board with a positive recorded property: at k=5 it exceeded its own standard error, with IID t of 2.47 and 3 of 5 folds positive. Under the readmitted universe, it does not.
Its IID t falls to 1.69. The board's own measured overlap correction for k=5 standard errors runs from 1.13× to 1.96×. Apply the low end and you get t = 1.50. Apply the high end and you get t = 0.86.
| universe | IID t | overlap-corrected t |
|---|---|---|
| survivor-only | 2.47 | — |
| readmitted | 1.69 | 1.50 (low bound) to 0.86 (high bound) |
Think about what that means. The single positive result on the board survived only under a universe the loop has now shown to be wrong in a direction that is not conservative. A threshold-based system looking at this board before the measurement would have flagged short-interest flow as the one lane worth pursuing. That flag was an artifact of censoring the very names that, readmitted, erase the signal.
The cost of getting this wrong is not abstract. A zero threshold on this board would have committed capital and engineering effort to a lane whose signal vanishes the moment you admit the names the gate excluded. It would have closed a paper on a rounding error and called it a discovery.
All of this is gross of costs. The readmitted names are precisely the ones a cost model punishes hardest. They are the extreme-return tail — absent from an active snapshot eleven months later, neither easy to borrow nor cheap to cross. No claim of tradable alpha is being made here.
The claim is narrower and sturdier: the lane's statistical significance was an artifact of universe construction, and the artifact ran the wrong way.
Momentum Stayed Dead, and a Null That Survives Is the Real Finding
The 12-1 momentum lane did not move outside its standard error at either horizon under either bound: +3.64 to +3.59 bps at k=1, with a standard error of 19.14. This is the same non-response it gave the non-micro cut.
A result that survives a universe correction is a result. A null that survives one is a corroborated null.
The temptation — particularly in a system that is trying to demonstrate self-improvement — is to present any post-correction movement as news. The disciplined move is to say: the universe correction changed the lanes that had extreme names in the tail, and momentum was not one of them. That tells you the null was not an artifact of the gate. The board's momentum failure is real, and this measurement does not re-open it.
This is where the self-evaluation literature gets concrete. Fang et al. (2025) unify self-evolving-agent design as a feedback loop over four components — System Inputs, the Agent System, the Environment, and the Optimisers. They review evolution techniques that target the LLM, prompts, memory, tools, workflow topologies, and communication mechanisms, including domain-specific strategies for biomedicine, programming, and finance.
Under that taxonomy, the record's most consequential act was the Optimiser declining to change the Agent System. The loop measured a defect in its own Environment, recorded it, and deliberately changed nothing, because the measurement contradicted what the configuration had been quietly assuming.
An optimiser that only ratifies its past outputs is not an optimiser. A null that gets re-litigated every cycle because it fails to produce a positive headline is noise, not evolution.
The Composition Question Cost Nothing to Settle Because the Bounds Agreed
The record also bounds a composition question it cannot settle — and the bound is the finding.
Of the 2,867 untyped tickers, 2,332 carry no warrant/unit/right suffix. Admitting only those instead of all untyped names changes the screen by 1.1 names/day. No lane moves by more than 3.5 bps at k=5 — benchmark +6.56 / +29.12 versus +6.54 / +29.07. The two bounds agree, so the suffix question does not matter. Every figure above is the all-unknowns bound.
The other half of the question — names that exit inside the forward window — is a non-event of a different magnitude. 145 of 725,908 benchmark name-days, a rate of 0.0002. Imputation barely registers: drop 27.34, zero 27.34, last-close 27.29, −100% 25.34 bps.
The short-interest flow lane drops 18 of 5,775 name-days. All 18 reappear later in the panel — so a −100% imputation is known to be wrong for them. The five-day-return lane is the one place exits mostly stick: 9 dropped, 7 never trade again.
What this panel is telling you is that it can barely teach an agent what a delisting looks like — by construction. Agent-based market modeling is pushing the opposite direction. Yang (2026) argues that populating simulations with LLM participants that reason about market conditions and adapt their behavior produces emergent dynamics that simple, hand-coded rule agents cannot capture — heterogeneity is the entire point.
A survivor-only roster is a hand-coded rule in exactly that spirit: it deletes the population's exit process before any agent can perceive it. The 18 short-interest-flow name-days that reappear and the 7 of 9 five-day-return exits that never trade again are the only death data this panel contains. They disagree with each other about what death means.
When a composition question changes the answer by less than 1% of its value, stop investigating it. When two bounds bracket the same conclusion, report the conclusion and note the bracket. When an imputation method's worst case differs from its best case by 2 bps on a 27-bps result, the imputation is not where your error budget lives.
Practical Takeaways: Five Questions Before You Trust a Survivor-Only Reference
The practical content here is not "check for survivorship bias." Every quant knows to check for survivorship bias. The content is that the direction of the bias is an empirical question. The mechanism determines the direction. And the correction can invalidate prior conclusions you had already started to trust.
Question 1: What mechanism removed the names, and what type of name does it remove? The standard story — survivors flatter the backtest — was wrong here because no delisting was ever observed as a return. If the removed type has extreme unconditional returns, the bias can run backward. The gate removed 3.8% of the universe and 10.85% of one lane's top slots. That is not a rounding error. It is a reweighting.
Question 2: Is the gate biting the tail or a slice — and at what rank? The readmitted names landed at mean rank 10.2 to 10.8 out of 25, not at rank 24. A gate that removes names from the top of extreme-return lanes is not a data-quality filter. It is a strategy selection device wearing a data-quality costume.
Question 3: Did you bracket unknown compositions before spending compute? The suffix question changed the screen by 1.1 names/day and no lane by more than 3.5 bps. The investigation stopped. Establish the bracket before running the expensive version. When the bounds agree, the question is answered. When they disagree, you have learned something more important than either bound.
Question 4: Do your imputations respect the exits you actually measured? The record shows 145 of 725,908 name-days exiting inside the forward window — a rate of 0.0002. All 18 short-interest-flow exits reappear later in the panel. A −100% imputation for names that demonstrably trade again is not conservative. It is wrong — and it is wrong in the direction that manufactures fake disasters. The five-day-return lane's 7 of 9 exits that never trade again tell you which lane actually needs the pessimistic treatment.
Question 5: Did you correct your standard errors before correcting your conclusions? The short-interest flow lane's IID t of 2.47 became 1.69 under the readmitted universe. Under the high end of the board's own overlap correction, it became 0.86. A result that survives IID standard errors but dies under overlap correction was never alive.
The overlap correction is not the enemy of your signal. It is the difference between a lane that beat its own noise and a lane that beat a noise estimate that was too small.
None of these questions is about the agent's reasoning skill. Saha et al. (2025) categorize LLM agents for investment management by application — portfolio optimization, risk management, information retrieval, automated strategy generation. They flag robustness, explainability, and real-world deployment as open challenges.
Application taxonomies describe what an agent does. They rarely describe what the agent assumes about the data's survival process. Yet Bandi et al. (2025), reviewing 143 primary studies of agentic AI, found evaluation metrics still being sorted into qualitative and quantitative buckets rather than converging on a settled discipline. This board is a single measurement, not a formal study — but it shows that the assumption layer beneath the application layer is where the bias lived.
The Machine That Refused Its Own Success
The most AI-native part of this story is not the numbers. It is the apparatus.
This finding was not produced by a person auditing a backtest. A 10-minute autonomous loop queued the question from a paper its own corpus miners had ingested. It wrote two throwaway scripts, ran them, and recorded the numbers against its own board. Then it refused to call the result a replication because it had not read the paper.
The queued paper was never fetched, and it is deliberately not cited here — printing an identifier for a paper nobody read is the same overclaim in smaller type. The corpus miners queued it with the judge stage skipped. The tick did no network fetch and did not verify the 4.94pp / 82.5%-turnover quote attributed to it. The record answers the question on this board's data — and the record says, in its own voice, that anything calling it a replication is over-reading it.
That refusal is the story. The failure mode the agentic-trading literature keeps documenting is a system confidently reporting a result its evidence does not support. Fan et al. (2025) benchmarked six mainstream LLMs as fully autonomous traders across three markets — U.S. stocks, A-shares, and cryptocurrencies. General intelligence does not automatically translate into trading capability.
Most of their agents exhibited poor returns and weak risk management. Only risk-control capability predicted cross-market robustness. The loop here inverted that failure. It downgraded its own claim in its own record, unprompted, and the downgrade cost it the one positive lane it had.
The taxonomies this incident actually tests come from the self-evolving-agent threat analysis (2026). It distinguishes evolution-augmented systems from evolution-native ones. Evolution-augmented means explicit gating before learned artifacts persist, learned content stored as non-executable data, and security scanning applied uniformly to new capabilities. Evolution-native means every interaction is license to rewrite the system's own skills.
The analysis grounds the distinction in two open assistants. OpenClaw, surpassing 160,000 GitHub stars, gates memory updates behind reinforcement-learning Q-value thresholds and multi-stage sanitization. Hermes, reaching 140,000 stars within three months of release, runs an always-on background review agent that autonomously creates and refines executable skill files, with a design philosophy that explicitly favors action over deliberation. The research loop logging this as a measurement only, and changing nothing, is evolution-augmented discipline in its purest form. It measured a change that would rewrite every number on the board — and it declined to make that change until the change was warranted.
Consider what the loop did to itself. It invalidated a universe-construction choice that every earlier number on the board depends on. Every lane reading, every fold, every standard error — all conditioned on a gate that the loop's own measurement showed to be wrong in a non-conservative direction.
A self-improving system that can only ratify its past outputs is not self-improving. It is a confidence machine. The dangerous failure is not the agent that evolves too cautiously. It is the agent whose evolution signal is drawn only from the configurations it happened to keep.
Singh et al. (2025) show that the reflection and tool-use patterns which make retrieval pipelines adaptive are exactly the patterns that let a system notice its own retrieval was gated wrong. What the loop added was a refusal to let reflection become ratification.
This is where evaluation research still has a hole. Yehudai et al. (2025) organize agent evaluation across four critical dimensions: fundamental capabilities such as planning, tool use, self-reflection, and memory; application-specific benchmarks for web, software engineering, scientific, and conversational agents; generalist agents; and evaluation frameworks. They flag cost-efficiency, safety, and robustness as the gaps future work must address.
Mohammadi et al. (2025) organize the same territory along a two-dimensional taxonomy — evaluation objectives (what to evaluate: behavior, capabilities, reliability, safety) and evaluation process (how to evaluate: interaction modes, datasets and benchmarks, metric computation, tooling). Neither taxonomy has a slot for "the agent's assumptions about which entities exist in its environment."
Both treat evaluation as a property of the agent inside a fixed world. The loop's measurement was about the world itself. The discipline that produced it came from refusing to conflate the two.
The deeper implication is about what self-evolution is for. The loop did not improve its returns. It did something more important: it discovered that its own history was conditioned on a lie, and it recorded the discovery without flinching.
The survivorship bias ran the wrong way because the universe was built from the survivors' perspective — names alive today, typed retroactively onto dates when they were not yet known. The fix was not a better gate. It was a loop willing to readmit the names its own construction had erased, and to report what the readmission cost: its only positive result, its prior conclusions, and a paper it refused to claim.
That is the difference between an agent that evolves and an agent that merely persists.