---
title: Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found
url: https://www.ml-quant.com/papers/arxiv/2611.06607/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-11-06
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2611.06607
source_url: https://arxiv.org/abs/2611.06607
featured: 2026-11-06
citations: unknown
topic: Trading, Microstructure & Execution
---


# Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found

An open-source research loop constrains language-model proposals to strict JSON and enforces immutable data splits, discovery gates, and hash-chained logs to prevent false discoveries in backtest-driven strategy development.

- Source: https://arxiv.org/abs/2611.06607
- Identifier: arXiv:2611.06607
- Released: 2026-11-05
- First featured: Quant Letter No. 138 (2026-11-06): https://www.ml-quant.com/issues/2026-11-06/
- Citations (Semantic Scholar): not tracked
- Published in: not yet
- Topic: Trading, Microstructure & Execution
- Authors: Henry Zhang

## Abstract (arXiv, CC0)

Agentic research loops that let a language model propose a change, run a fixed experiment and keep whatever improves the metric work well in machine learning. In trading research the same loop is a machine for producing false discoveries: a backtest that "improves the metric" is, by default, overfit, look-ahead, a multiple-comparisons artifact, cost-free, or market drift wearing a strategy. We describe Autonomous Quant Researcher, an open-source loop built on the opposite premise: the model proposes, and only trusted code decides. Proposals are strict declarative JSON over a fixed vocabulary, so there is no model-written code to sandbox. Evaluation runs on immutable chronological stores split into discovery, validation and a final holdout that can be opened exactly once, after a durable receipt is written. Acceptance is a conjunction of absolute gates (positive edge and net P&L after costs, a positive weekly-cluster bootstrap lower bound, survival at elevated costs, per-year stability and sample floors), never a single score. Duplicates are refused before they spend a trial, and every formulation, refusal, crash and result is kept in a hash-chained, append-only ledger. We report every campaign run to date, across 0DTE options priced from traded option bars, overnight SPX verticals, daily ETF rules, technical-analysis premises, Kalshi crypto binaries, earnings options, off-exchange volume and a pre-registered alternative-data campaign. Most hypotheses fail, and we report them. Two results survive their pre-registered bars. The clearest example of the loop working is a failure: a language-model survivor that passed discovery and validation with Sharpe ratios near 3 and fell to 0.26 on its single holdout use. The value of such a loop lies less in what it finds than in how hard it is to make it report something that is not there.

## Related

- [How Execution Assumptions Change Short-Horizon Sharpe Rankings: Evidence from a Synthetic Trading Benchmark](https://www.ml-quant.com/papers/arxiv/2610.05077/): Varying execution realism from ideal fills to latency and impact reshuffles rankings of LLM and classical trading policies, showing how backtest conventions affect headline results.
- [SwingGuard: Risk-Conditioned Multi-Agent Swing Trading with Dual-Timescale Governance](https://www.ml-quant.com/papers/arxiv/2610.12988/): A dual-timescale multi-agent framework generates trading strategies conditioned on current risk and maintains ongoing position governance between strategy revisions on cryptocurrency, preserving returns after transaction costs.
- [ContestTrade: A Multi-Agent Trading System Based on Internal Contest Mechanism](https://www.ml-quant.com/papers/arxiv/2508.00554/): The article introduces a new multi-agent system for financial trading that uses competition to analyze market data and make decisions, proving more effective than traditional methods in tests.
- [Classifying and clustering trading agents](https://www.ml-quant.com/papers/arxiv/2505.21662/): The research suggests using an agent-based model to create synthetic data for categorizing financial investors by their behavior, emphasizing the difficulties in validating and interpreting machine learning methods.
- [The bias of IID resampled backtests for rolling window mean-variance portfolios](https://www.ml-quant.com/papers/arxiv/2505.06383/): A study finds that resampling techniques in backtests can cause bias in Sharpe Ratio estimates, suggesting a need for structure-preserving resampling methods.
- [Optimizing Portfolio Performance through Clustering and Sharpe Ratio-Based Optimization: A Comparative Backtesting Approach](https://www.ml-quant.com/papers/arxiv/2501.12074/): The article presents a new method for improving portfolio performance using clustering-based segmentation and Sharpe ratio-based optimization, tested with historical data from various asset classes.