Thinking Fast, Trading Late: System One Decision Models vs the Market

Large language models (LLMs) read company news at scale, and backtests suggest that their readings predict stock returns. We ask how much of that predictability survives a test that rules out look-ahead, memorization and selection. We propose OLIVIA, a protocol for such tests: point-in-time inputs with company identities masked, selection on validation data only, an analysis plan frozen by cryptographic hash, and a test period unblinded once through a logged lock. We apply it to AI decision models, non-generative language models that return probabilities for typed questions, reading news about S&P 500 stocks: an open-weights model we fine-tuned on 2024-2025 data, three commercial models and standard baselines. The fine-tuned model's signal replicated on two validation quarters (daily information coefficients 0.055 and 0.065) but fell to 0.020 (t = 1.7) on the blinded January-July 2026 test. The commercial models' readings line up with the overnight move to the open (information coefficient 0.074 to 0.077) but not with the move after it, so the news is priced by the time a trader can act on it at the open. No model can tell over- from under-reaction, and no pre-registered strategy survives costs and a deflated Sharpe ratio. Commercial models identify masked companies in 62% to 74% of cases against a 28% baseline: masking hides names, not companies. The market reads the news first.

1. 14 May 2025

   Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?

   FINSABER, a backtesting framework, shows that Large Language Models' effectiveness in stock trading decreases over longer periods and larger symbol universes, emphasizing the need for trend detection and risk controls.

   arXivIn Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1

   29cites
2. 23 Apr 2025

   The Memorization Problem: Can We Trust LLMs'Economic Forecasts?

   The study shows that large language models can remember exact economic figures from before their knowledge cutoff dates, which may skew their predictive abilities in forecasting and backtesting trading strategies.

   arXiv

   43cites
3. 6 Nov 2026

   LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

   A benchmark of 2,226 dated facts reveals that frontier language models underreport their knowledge and answer most facts they claim not to know, compromising backtest validity.

   arXiv

   4fanfare
4. 9 Oct 2026

   MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

   Introduces a benchmark that compares LLM-generated trading code to reference strategies bar by bar on identical data, revealing silent failures in implementation despite passing functional tests.

   arXiv

   0cites
5. 4 Oct 2023

   Look-Ahead Bias in Stock Return Predictions

   Large language models like ChatGPT can generate profitable trading signals from news sentiment, but backtesting can yield biased results due to overlapping periods.

   SSRN

   5shares
6. 7 Aug 2024

   Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

   The research investigates enhancing Large Language Models' (LLMs) performance using more test-time computation, suggesting a compute-optimal scaling strategy based on prompt difficulty.

   Machine learning

   2,390cites