The hallucinated edge
Why AI trading agents produce extraordinary backtests and ordinary losses: the general mechanics of model hallucination, the four leakage channels specific to markets, and the information-set audit that closes them.
CUBICLE, BY ALGOTRADA · ON TAIFOON RAILS · 2026 · V1.0
1 · Hallucination is not a bug in the model. It is the objective.
A language model is trained to minimize next-token loss: to produce the most plausible continuation of a context, weighted by everything it has read. Plausibility and truth usually correlate in language (that is why the models feel intelligent), but the objective never references the world, only the corpus. When the corpus is silent, the model does not report silence. It interpolates. The output arrives with the same fluent confidence either way, because confidence is a property of the generated text, not a property of the model’s knowledge.
Three mechanics turn this into what practitioners call hallucination. First, memorization blurs into generalization: training compresses the corpus into weights, and a recalled fact and a fabricated one are produced by the same sampling machinery, so the model itself cannot reliably tell you which it did. Second, there is no calibrated uncertainty: token probabilities are not probabilities about the world, and post-training with human feedback makes it worse, because raters reward confident, agreeable prose. Third, rationales are narratives, not traces: when a model explains its answer, it is generating a plausible story about a computation, not printing the computation. The explanation can be eloquent and wrong.
None of this matters much when the task has a grader. Code compiles or it doesn’t; the theorem checks or it doesn’t. It matters enormously when the grader is slow, noisy, and expensive. There is exactly one domain where the grader is the future itself.
2 · Markets are the worst case for these failure modes
Trading amplifies every general weakness above, and adds four leakage channels that inflate a backtest while looking, from the outside, like skill.
Leak 1: The memorized future
An LLM trained through 2024 and backtested on 2019–2024 has already read that tape. Not necessarily the candles, but the narrative: “the March 2020 crash”, “the 2023 AI rally”, ten thousand recaps, papers and threads that describe what happened next. When it “predicts” inside its training window, it is recalling, dressed as inference. This channel is unique to LLMs. A rules engine physically cannot know the future; a language model has read a compressed version of it. It is the single biggest reason AI-agent backtests look extraordinary, and it invalidates any in-corpus evaluation window regardless of how carefully prices are hidden. The direction leaks through language.
Leak 2: Selection on noise
Markets have brutally low signal-to-noise: daily direction is a coin flip with a hairline bias. In that regime, run two hundred prompt or parameter variants and the best one will look brilliant by construction. You have selected the maximum of two hundred noisy draws, not a strategy. Quant finance knows this disease (backtest overfitting, the deflated Sharpe ratio); prompt space makes it worse because variants feel qualitative, cost nothing, and nobody registers how many were tried. The winning prompt is the luckiest sample, and the search that found it is invisible in the result.
Leak 3: Fill fantasy
Simulators without an order book assume fills at mid, with no queue position, no slippage, no partials and no impact. Every one of those assumptions transfers money from reality to the backtest. Worse is the adverse-selection asymmetry of limit orders: in a real book, your resting bid is filled most reliably precisely when the market is going lower, so the fills you model as wins are enriched for losses. At high turnover, spread plus impact per trade frequently exceeds the entire claimed edge. The strategy did not have alpha; the simulator had generous fills.
Leak 4: One regime
A price series is one draw from a process that keeps changing: volatility regimes, correlation flips, policy eras, market-structure changes. An agent tuned on a single window learns that tape’s personality: its dip-buying rhythm, its mean-reversion half-life, its session behavior. That is memorization at the distribution level, and it fails on contact with a different regime. Add reflexivity (deploying the strategy changes the market it trades) and the tape you tested is not even the tape you will trade.
Underneath all four sits the evaluation asymmetry: language tasks get graded in seconds; a trading claim is graded by months of live capital. So the industry optimizes the proxy (the backtest) and Goodhart’s law does the rest. The model produces a confident narrative; the leaks make the numbers agree with it; and nobody in the loop can reconstruct what the agent actually knew at decision time.
The hallucination is not an invented price. It is an invented information set.
3 · The wedge: deployment is outrunning evidence
Agent order flow keeps growing (exchange agent APIs, agentic wallets, token-funded agent operators) while the share of that flow with a reproducible, verifiable record stays flat. Capital is being allocated into systems nobody can audit, on track records nobody can reconstruct. Regulators are converging on the same gap from the other side: mandatory AI-decision explainability is arriving faster than the tooling that could satisfy it. The widening area between those two curves is the market for evidence.
4 · Closing the leaks: audit the information set
Every existing trust mechanism audits results: broker-verified track records, staked predictions, on-chain fills. All post-trade. The four leaks live pre-trade, in what the model could know and how the winner was selected. Closing them requires controlling the information set itself:
Against the memorized future, evaluate on tape the model has not read: post-training-cutoff windows, replayed live. The clock owns the data; a candle exists for the agent only after dispatch. The receipt proves the sequence.
Against selection on noise, sessions are first-class objects: every variant run is recorded, so the denominator is visible. A published strategy carries its full session history, not its best run.
Against fill fantasy, decisions execute against a real central limit order book with resting orders, queue, partials and fees. Paper capital, real microstructure.
Against one regime, the same strategy replays across many windows and markets at up to 10,000×; regime coverage becomes a property of the receipt, not a claim in a pitch.
# the receipt, concretely session sid_7f2a · NQ · 2026-05-01..05-30 · speed 100× ticks dispatched 41,760 gaps 0 decisions 27 each stamped: causal bar + seed hash fills 27 book-real, queue-aware causality_ok TRUE · orphan_decisions 0 · pnl +3.1R three actors cross-checked: dispatcher · strategy · book
5 · Two vantage points, one discipline
Evidence as an operator discipline
We run our own engines (Skydweller on index futures) under the same regime we sell: no strategy of ours ships a number that is not receipted. The Cubicle is that discipline productized: a kitchen where any agent’s recipe is cooked under a clock it cannot cheat, and the audit trail is the deliverable. A trading claim without its receipt is testimony; we are in the evidence business.
Determinism as infrastructure
Rails do not have opinions about strategies; they have invariants. One matching engine, one replay clock, one cockpit rendered from one core. The same code path that feeds the agent feeds the auditor. Receipts are a protocol primitive, not a report: any venue skin on these rails (Cubicle is the first) inherits them, and any future skin (a fund’s internal desk, a competition, an exchange) gets the same property for free.
6 · What this makes possible
When the information set is auditable, things that are currently marketing become markets. A strategy listing with receipts is underwritable, so subscribers can price it. An agent’s training run is a corpus of receipted episodes of read, decision and outcome that reinforcement learning can grade honestly. And a regulator’s explainability requirement becomes a query, not a committee. The proving ground comes first; everything downstream (the marketplace, the staged live rails, the open trading model trained on the receipted corpus) is compounding interest on the same primitive.
Backtests should be evidence. Until they are, they are advertising.