arXivML & AI Methods
An AI-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments
An AI-powered replication of 324 papers reusing natural experiments finds only 25% of defensible specifications preserve published sign and significance, quantifying dependence on analysis choices.
Featured in No. 140 on 20 Nov 2026 · on release day

- Released
- 20 Nov 2026
- First featured
- No. 140 · 20 Nov 2026
- Published in
- Not yet, as far as Semantic Scholar knows
- Fanfare
- 4 of 5
- Identifier
- arXiv:2611.25184
- Authors
- Ruijiang Gao and Steven Chong Xiao
Abstract
From arXiv (CC0).
Empirical finance research often reuses natural experiments to study other outcomes, raising concerns about multiple testing and publication bias. We use AI agents to run a multiverse analysis of 324 papers that reuse 13 natural experiments. The agents rebuild the data, reproduce each paper's main estimate and re-estimate it across grids totaling 62,409 estimated defensible specifications. Based on the reproductions and the specification grids, we develop measures of empirical robustness. We reproduce 42% of the main estimates of claims with a signed effect under the papers' own specifications. Across all defensible specifications, on average 25% keep the published sign and have $p < 0.10$. Across experiments, claims based on regression discontinuity are more robust than difference-in-differences claims. Within experiments, claims about first-order effects, claims with larger published $t$-statistics, claims whose authors' data or code we find, claims in top journals, and claims with fewer earlier papers on the same experiment are more robust. Claims on experiments that more papers reuse are not less robust on average. Our findings quantify how much published claims depend on analysis choices, thereby showing that large-scale multiverse analysis becomes feasible with agentic coding.
Citations and venue from Semantic Scholar (ODC-BY), refreshed weekly. Summary: Quant Letter (CC BY 4.0).