---
title: An AI-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments
url: https://www.ml-quant.com/papers/arxiv/2611.25184/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-11-20
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2611.25184
source_url: https://arxiv.org/abs/2611.25184
featured: 2026-11-20
citations: unknown
topic: ML & AI Methods
---


# An AI-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments

An AI-powered replication of 324 papers reusing natural experiments finds only 25% of defensible specifications preserve published sign and significance, quantifying dependence on analysis choices.

- Source: https://arxiv.org/abs/2611.25184
- Identifier: arXiv:2611.25184
- Released: 2026-11-20
- First featured: Quant Letter No. 140 (2026-11-20): https://www.ml-quant.com/issues/2026-11-20/
- Citations (Semantic Scholar): not tracked
- Published in: not yet
- Topic: ML & AI Methods
- Authors: Ruijiang Gao, Steven Chong Xiao

## Abstract (arXiv, CC0)

Empirical finance research often reuses natural experiments to study other outcomes, raising concerns about multiple testing and publication bias. We use AI agents to run a multiverse analysis of 324 papers that reuse 13 natural experiments. The agents rebuild the data, reproduce each paper's main estimate and re-estimate it across grids totaling 62,409 estimated defensible specifications. Based on the reproductions and the specification grids, we develop measures of empirical robustness. We reproduce 42% of the main estimates of claims with a signed effect under the papers' own specifications. Across all defensible specifications, on average 25% keep the published sign and have $p < 0.10$. Across experiments, claims based on regression discontinuity are more robust than difference-in-differences claims. Within experiments, claims about first-order effects, claims with larger published $t$-statistics, claims whose authors' data or code we find, claims in top journals, and claims with fewer earlier papers on the same experiment are more robust. Claims on experiments that more papers reuse are not less robust on average. Our findings quantify how much published claims depend on analysis choices, thereby showing that large-scale multiverse analysis becomes feasible with agentic coding.

## Related

- [Agentic AI in Automated Markets](https://www.ml-quant.com/papers/arxiv/2610.22522/): Deep reinforcement learning simulates three agent archetypes in automated markets, finding herd-followers generate frequent short stress episodes while manipulators create persistent volatility.
- [Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors](https://www.ml-quant.com/papers/arxiv/2609.27051/): Proposes a statistical referee that judges investment factors proposed by language-model agents using out-of-sample market outcomes, ensuring false-discovery control at any stopping time.
- [AlphaDiverse: Post-Training Local Quantitative Research Agents for Diverse Exploration in Alpha Factor Mining](https://www.ml-quant.com/papers/arxiv/2609.29014/): Proposes a multi-agent system with post-training that automates alpha factor mining locally, using diverse research paths and joint optimization to broaden exploration while maintaining prediction quality.
- [Bounded Normative Equivalence in Human-AI Cooperation: Group Behaviour, Not Partner Labels, Predicts Cooperation under Anonymous Aggregate Feedback](https://www.ml-quant.com/papers/arxiv/2601.20487/): AI agents in group settings can influence cooperation just like humans do, showing that social norms can adjust for both AI and human participants.
- [The Adoption and Usage of AI Agents: Early Evidence from Perplexity](https://www.ml-quant.com/papers/arxiv/2512.07828/): A study of AI agent use with the Comet browser reveals that personal productivity and learning are the main reasons users interact with these tools.
- [General Social Agents](https://www.ml-quant.com/papers/arxiv/2508.17407/): Modern AI agents can be used to apply social science theories to new settings with little or no modification, and have been found to predict human behavior more accurately than traditional methods in a sample of new games.
