---
title: LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
url: https://www.ml-quant.com/papers/arxiv/2611.06808/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-11-06
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2611.06808
source_url: https://arxiv.org/abs/2611.06808
featured: 2026-11-06
citations: unknown
topic: LLMs & Text
---


# LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

A benchmark of 2,226 dated facts reveals that frontier language models underreport their knowledge and answer most facts they claim not to know, compromising backtest validity.

- Source: https://arxiv.org/abs/2611.06808
- Identifier: arXiv:2611.06808
- Released: 2026-11-05
- First featured: Quant Letter No. 138 (2026-11-06): https://www.ml-quant.com/issues/2026-11-06/
- Citations (Semantic Scholar): not tracked
- Published in: not yet
- Topic: LLMs & Text
- Authors: Lingfei Qian, Yan Wang, Xueqing Peng, Yupeng Cao, Yuechen Jiang, Zhuohan Xie, Anke Xu, Wenbo Cao, Jiarui Chen, Zehan Li, Jimin Huang, Alejandro Lopez-Lira

## Abstract (arXiv, CC0)

Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.

## Related

- [Interpreting the Interpreter: Can We Model post-ECB Conferences Volatility with LLM Agents?](https://www.ml-quant.com/papers/arxiv/2508.13635/): A new method using a Large Language Model can predict financial market responses to European Central Bank press conferences, aiding in maintaining financial stability.
- [Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?](https://www.ml-quant.com/papers/arxiv/2505.07078/): FINSABER, a backtesting framework, shows that Large Language Models' effectiveness in stock trading decreases over longer periods and larger symbol universes, emphasizing the need for trend detection and risk controls.
- [The Memorization Problem: Can We Trust LLMs'Economic Forecasts?](https://www.ml-quant.com/papers/arxiv/2504.14765/): The study shows that large language models can remember exact economic figures from before their knowledge cutoff dates, which may skew their predictive abilities in forecasting and backtesting trading strategies.
- [Thinking Fast, Trading Late: System One Decision Models vs the Market](https://www.ml-quant.com/papers/arxiv/2611.03985/): A pre-registered blinded test finds that language-model readings of company news lose their predictive power in real time and that commercial models identify masked companies in two-thirds of cases.
- [MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code](https://www.ml-quant.com/papers/arxiv/2610.03080/): Introduces a benchmark that compares LLM-generated trading code to reference strategies bar by bar on identical data, revealing silent failures in implementation despite passing functional tests.
- [LLM-Based Semantic Surprises in FOMC Communication: Asset Prices and Financial-Market Stress](https://www.ml-quant.com/papers/ssrn/7519200/): Semantic surprises extracted from Federal Reserve statements predict subsequent financial-stress dynamics and reduce forecast error by up to 23%, particularly when initial stress is high or during recessions.