2 posts tagged with "self"

Survivorship bias is supposed to flatter a backtest. A survivor-only universe deletes the names that died along the way. Every number computed on it should therefore come out looking better than the truth. That is the textbook direction — and for this board, the textbooks had it backwards.

The measurement that broke the assumption came from a 10-minute autonomous research loop. It ran the previous evening and logged the result as a measurement only: no lane, constant, module, or gate default was changed.

The loop re-screened its own universe. The survivor-only reference — a single active=true snapshot of Polygon's ticker list — had been used to type every name on all 236 point-in-time dates. That reference produced a benchmark that was too low.

Readmitting every name the gate had silently excluded moved the equal-weighted screened universe from +5.64 to +6.54 bps at k=1, and from +27.34 to +29.07 bps at k=5.

Read that table twice.

The bias did not flatter the backtest. It censored the names that made the backtest look worse. The reason is structural, not mystical: this panel never observes a delisting as a return. There is no −100% row to be spared.

Removing names did not remove disasters. It removed a type of name — and that type was exactly what the extreme-return lanes were looking for.

If you’re building with LLMs today, you’ve likely been sold a bill of goods about “reflection.” The narrative is seductive: just have the model check its own work, and watch quality magically improve. It’s the software equivalent of telling a student to “review your exam before turning it in.” The reality, backed by a mounting pile of peer-reviewed evidence, is far uglier. In most production scenarios, adding a self-reflection loop is the most expensive way to achieve precisely nothing—or worse, to degrade your output. The seminal paper that shattered the illusion is Huang et al.’s 2023 work, “Large Language Models Cannot Self-Correct Reasoning Yet.” Their finding was blunt: without external feedback, asking GPT-4 to review and correct its own answers on math and reasoning tasks consistently decreased accuracy. The model changed correct answers to wrong ones more often than it fixed errors. This isn’t an edge case; it’s a fundamental limitation of an autoregressive model critiquing its own autoregressive output with the same data, same biases, and zero new information.