AI-first retrieval for trading: when BM25 beats the embedding you were going to reach for
BM25 outperforms dense retrieval when queries contain rare, exact identifiers — tickers, CIK numbers, dates, or uncommon financial terms — where lexical matching is more precise than semantic similarity, especially on short, domain-specific documents.
The 2026 reflex when new market text lands is now automatic. Embed it. Vector-index it. Let semantic similarity answer the questions. This system did none of that, because the architecture banned the embedder — not because a retrieval bake-off said so, but because a safety rule priced the cost at infinity.
All figures were measured 2026-09-07 on the production box. The result is worth stating bluntly: the ban produced a better index than free choice would have.
The architecture banned the embedder, and the index improved
The system separates code into planes. The dependency graph enforces the separation, not code review. One rule matters here: no model-bound call may sit on a path that reaches order submission, and the engine holding a model must be unimportable from the execution plane.
A neutral component inherits both bans. Anything neutral code depends on is inherited by every plane depending on it. Data acquisition is neutral by nature: both sides need it, neither may own it.
So the new ingest component could not link an embedding backend. Not "should not." The build fails. Resolving the dependency tree against the ban list found 1,878 crates for the vector store and zero forbidden prefixes — and only because default features were off. The default feature set pulls embedding backends and would have failed the build.
Worth measuring against the field's reflexes: Xia et al. (2026) coded a protocol snapshot of 77 studies on LLM trading agents screened through 2026-03-09. Within their primary empirical subset of 19, only 2/19 report extractable time-consistent split protocols, 1/19 documents a transaction-cost model, and no study reaches R3 reproducibility. Architecture experimentation is expanding fast; comparable evaluation is the bottleneck.
A system that reflexively reaches for an embedding model is making the same trade unexamined. Here the dependency graph reached the conclusion first.
A ticker is an identifier, not a word
The corpus is about 8,700 rows of market news, regulatory disclosures and insider filings. Those are the places where exactness is not a preference. It is the meaning.
CAR is Avis Budget. Embed it and it lands near automobiles, vehicles and driving. Match it exactly and it is a company.
Dense retrieval is built to collapse surface differences into shared meaning. That is precisely the wrong behaviour for a symbol whose meaning is its surface. The same holds for accession numbers, CUSIPs and the legal phrases that separate one disclosure category from another.
Retrieval, unedited: query "short squeeze" returned "With Nearly 40% Short Interest, Is SoundHound Stock a Buy or Sell?" Query "insider purchase" returned three exact filings, including "Founder and Chairman Marcos Mindlin Buys 1.3 Million Shares."
Those are exact hits because the content is tickers, proper nouns, numbers and legal phrases. Paraphrase-matching is what dense retrieval buys. A filing does not paraphrase.
The same is not a corner case in agentic finance. Zhu et al. (2026) built KTD-Fin precisely because long backtests overlap model knowledge cutoffs. Their data-side protocol anonymizes key identifiers and calendar information across ten frontier LLM agents evaluated on the Chinese CSI300 over a 2024–2026 window. Masking substantially changed agent rationales, pushing them toward anonymized factor-based reasoning.
The implication for retrieval is mine, not theirs: if an identifier changes what an agent can reason about, then the retriever that approximates identifiers is corrupting the input before the model sees it.
The number in the headline is the signal — embeddings smear it
"With Nearly 40% Short Interest" is a tradeable fact because of the 40. A retriever tuned for paraphrase treats that headline as near-identical to one saying "elevated short interest." Those are not the same trade. Lexical scoring keeps the number attached to the claim.
This is where grounding stops being a retrieval abstraction. Jiang et al. (2026) evaluated four leading LLMs on InvestLogicBench, a process-native benchmark of 201,247 documented decisions from 151 real-world investors. Logical plausibility stayed near 4/5, but event grounding scored only 0.8–2.8/5.
That gap — polished reasoning built on weakly grounded events — is what outcome-only evaluation hides. Look at where the grounding comes from in an agentic pipeline. It comes from retrieval.
The production corollary is sharper. If the ingest index preserves the exact token, the event stays attached to the claim. If it embeds, the number becomes a smear and the reasoning above it has nothing to stand on.
The second run was the measurement, not the first
The ingest was rewritten to write straight into the store from the fetching process, with no serialisation boundary. First run: 8,671 rows in 1.1 s, or 8,196 rows/s. Second run, same window: 8,671 rows in 0.5 s, or 17,665 rows/s.
The second run is the important one — and not because it is faster. It returned the identical row count.
The first version of the upsert key was not unique. One regulatory filing produces several disclosure rows sharing a filing URL. A re-run would have silently doubled the table.
Only running it twice could reveal that. The first run cannot tell you your key is wrong.
That is a measurement lesson, not a performance note. A benchmark run once is not a benchmark. It is an assertion with a timestamp. The habit of treating a single run as the verdict is exactly the reproducibility gap the trading-agent literature keeps hitting: Xia et al. (2026) found protocol incomparability across their 19 primary studies, with evaluation protocols, execution semantics and reproducible artefacts flagged as the field's immediate bottlenecks.
The fix cost 0.5 s and one rerun. The bug it caught would have corrupted every downstream claim.
The rate limit was imaginary and it cost four hours
The fetcher was sleeping 13 seconds between calls to honour a rate limit from a plan the account no longer had. The vendor never throttled once.
Same endpoint, same machine, same minute. Every row is zero rate-limit responses:
| client | concurrency | throughput | rate-limited |
|---|---|---|---|
| threads | serial | 23.6 req/s | 0x 429 |
| threads | 32 | 511 req/s | 0x 429 |
| threads | 128 | 1,072 req/s | 0x 429 |
| async, native | 32 | 1,192 req/s | 0x 429 |
| async, native | 128 | 3,721 req/s | 0x 429 |
| async, native | 256 | 3,638 req/s | 0x 429 |
| async, native | 512 | 5,657 req/s nominal | 0x 429, 1,485 of 2,048 completed |
Two things fall out. Latency stayed flat at 37 ms median through every width — the signature of a service that is not queueing you. A throttled endpoint slows down before it refuses.
Second: 128 in-flight requests at 37 ms implies a ceiling near 3,400 req/s. The threaded client was leaving roughly two-thirds of available throughput unused. The interpreter was the limit, not the network and not the plan.
The 512 row is not a win. Nothing was rate-limited there either — the client dropped 563 requests of its own. The honest ceiling is 128 to 256. Reporting 5,657 req/s from that row would be reporting a failure as a record.
The cost of the old constant: the full history took 4.1 hours under the 13-second sleep and 38 seconds without it. Live benchmarks exist for exactly this reason. Qian et al. (2025) built Agent Market Arena as a continuous, real-time multi-market benchmark because fixed windows and limited periods let evaluation hide behind unverified assumptions.
The quiet damage was statistical. Five calls a minute could buy 530 days of history, so the system's headline tail measurement rested on 231 dates. Over a thousand are now reachable in under a minute. The constraint had been choosing what could be measured all along.
The better engine lost on integration cost
A graph-vector database was assessed for the same job. It is the lighter and arguably more interesting engine — 312 crates against 1,878 — and the data genuinely is a graph: insider to company, company to filing category, company to related company.
It was declined on one criterion. A table nothing reads is worthless.
The vector store already in use is the one every existing reader queries. A second store would need new code on both sides before a single question could be asked of it. Two stores holding the same rows are two sources of truth.
The trigger for revisiting is written down rather than left to taste: adopt it when there is a graph question actually being asked — "insiders who bought at companies that also filed a material-agreement disclosure the same week" is the shape. Then, alongside, for traversal. Not before.
This is the harder version of the argument. The ban was not making the system dumber by excluding a better engine. It was forcing the deployment decision onto integration cost — the one cost that compounds.
Squeeze scoring is null until retrieval supplies the conditional
Now the part that makes this a trading decision and not an infrastructure note.
The same system scored two classical squeeze metrics as cross-sectional ranking lanes over 231 dates at full coverage. Short interest relative to free float. Days to cover. Both are nulls: the volume-normalised one at +5.00 bps against a standard error of 12.35, and the float-normalised one negative at the weekly horizon.
That is the honest state of the static setup. Knowing a name is heavily shorted does not, by itself, rank names that outperform.
The squeeze premise, if it survives anywhere, survives as a conditional one: heavily shorted and something happened. The something is a filing, an insider buy, a disclosure. It lives in exactly the corpus this index was built over.
The parallel to agent attribution is uncomfortable. Zhu et al. (2026) found that under leakage-controlled evaluation, LLM agents' cumulative returns were largely explained by passive market and style exposure, with limited evidence of persistent stock-selection alpha. The static signals in our own system show the same shape — nothing left when the conditioning information is removed.
Do not over-read this. The conditional version has not been measured. What has been measured is that the unconditional version does not work, and that the data needed to test the conditional one is now indexed and searchable.
Retrieval is what supplies the why. A screen built on price and volume can see a name move 50% in a week and never see why. Whether the system can find the why depends on whether the retriever treats a ticker as a symbol or as a smear of related concepts.
Five questions before you embed a corpus
The honest boundary is this: BM25 is not universally better. The same system keeps a dense index on its research-paper corpus, and that is the right call there. Prose questions share meaning without sharing words; dense retrieval earns its place. The decision is corpus shape, not fashion.
Why does BM25 beat embeddings on financial text? Financial language is identifier-dense: tickers, CUSIPs, accession numbers, dates and legal phrases. Lexical matching preserves those exactly; embeddings approximate them. Semantic similarity is the wrong operation when surface form is the meaning.
When should I choose BM25 over dense retrieval for trading data? When queries are dominated by exact identifiers and numeric claims — tickers, instrument codes, versioned regulatory text. Choose dense when the corpus is paraphrase-heavy: earnings-call summaries, analyst narratives, research prose.
Can I combine BM25 and embeddings in one system? Yes. Hybrid architectures run both indexes and merge results with reciprocal rank fusion or weighted scoring. The cost is real: an extra index, extra latency, and every row embedded before it is searchable. Hybrid earns its keep only when the query mix is genuinely split between identifier queries and paraphrase queries.
How do I know which retriever works? Test both against labelled queries from production. Measure P@k, recall@k and latency under load. The live-measurement discipline matters more than the algorithm choice. Fan et al. (2025) evaluated six LLMs across U.S. stocks, A-shares and cryptocurrencies in AI-Trader and found general intelligence does not automatically translate into trading capability — behaviour is benchmark-specific.
What are the real limits of embedding-based retrieval? Polysemy and symbol collision. call, put, short, float, spread mean different things by context, and a ticker can collide with an ordinary word. The model version also silently changes what "similar" means, forcing a re-embed of the whole corpus each time.
The architecture priced the embedding at infinity. Free choice would have spent weeks comparing indexes. The constraint reached the right answer faster, and the measurement discipline caught an idempotence bug and a phantom rate limit on the way.
The next vector store gets adopted when a graph question is actually being asked. Not before. The next embedding model gets a path when the corpus is prose, not filings. The index should match what a trader must not lose: the number, the identifier, the exact event.
A safety rule chose the unfashionable index and the system got faster, more reproducible and more honest about what it cannot yet claim. Sometimes the best retrieval decision is the one you were forbidden from making.