---
title: What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books
url: https://www.ml-quant.com/papers/arxiv/2610.20104/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-10-16
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2610.20104
source_url: https://arxiv.org/abs/2610.20104
featured: 2026-10-16
citations: unknown
topic: ML & AI Methods
---


# What Do Foundation Models for Market Microstructure Learn? A Controlled Study with Synthetic Limit Order Books

Autoregressive transformers trained on limit-order-book message streams learn amortized Bayesian inference over market regimes, matching optimal prediction and enabling accurate parameter calibration.

- Source: https://arxiv.org/abs/2610.20104
- Identifier: arXiv:2610.20104
- Released: 2026-10-15
- First featured: Quant Letter No. 135 (2026-10-16): https://www.ml-quant.com/issues/2026-10-16/
- Citations (Semantic Scholar): not tracked
- Published in: not yet
- Topic: ML & AI Methods
- Authors: Fernando Acero

## Abstract (arXiv, CC0)

Autoregressive transformers trained on limit order book (LOB) message streams are increasingly proposed as foundation models for market microstructure. Because the generative process of real markets is unknown, these models can be judged only by the realism of their output, leaving open what they actually learn. We make the question answerable by inverting the setup: we train GPT-style models on order flow from simulators whose latent parameters are known, so that the Bayes-optimal predictor, the generating state, and the exact data entropy serve as references. Measured against them, next-token pretraining on order flow implements amortized Bayesian inference over market regimes. The learned predictive matches the exact Bayes optimum to within a millinat per token and approaches the oracle floor even on a nonstationary family whose latent state drifts within an episode, a regime that requires filtering. The latent regime is linearly decodable from frozen activations, well above baselines, so a frozen-model read-out calibrates simulator parameters more accurately than windowed maximum likelihood at a fraction of the cost. In our experiments these capabilities are not produced by scale: capacity and token volume saturate early, and it is the diversity of training regimes, not the token count, that determines generalization to unseen markets. On real NASDAQ data, zero-shot transfer is set by how well the simulator prior covers real flow: an enriched mixture nearly closes the gap to an in-domain baseline, and pretraining beats from-scratch training at every budget across held-out tickers. Known ground truth turns average next-token loss, which is uninformative near the entropy floor, into an exact comparison against the Bayes-optimal and oracle predictors.

## Related

- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://www.ml-quant.com/papers/arxiv/2501.01423/): The paper proposes a new model, VA-VAE, that aligns the latent space with pre-trained vision foundation models, enabling faster convergence of Diffusion Transformers in high-dimensional latent spaces and achieving top performance on ImageNet 256x256 generation.
- [Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models](https://www.ml-quant.com/papers/arxiv/2411.04996/): The Mixture-of-Transformers (MoT) is a sparse multi-modal transformer architecture that reduces pretraining costs and allows modality-specific processing with global self-attention.
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://www.ml-quant.com/papers/arxiv/2408.05178/): ECG-FM, a transformer-based model for ECG analysis, shows strong performance in predicting cardiac conditions, having been pretrained on 2.5 million samples.
- [Jamba-1.5: Hybrid Transformer-Mamba Models at Scale](https://www.ml-quant.com/papers/arxiv/2408.12570/): Transformer-Mamba Models: Jamba-1.5 is a new large language model with enhanced conversational and instruction-following capabilities, featuring a unique quantization technique for cost-effective inference.
- [KiT: A Foundation Model for Financial Time-Series Forecasting using DiffusionTransformers](https://www.ml-quant.com/papers/arxiv/2609.34507/): Diffusion transformer foundation model reformulates candlestick prediction as conditional trajectory generation, achieving 0.057 mean return RankIC across markets and timescales.
- [Transformers in AI Research](https://www.ml-quant.com/papers/ssrn/4945556/): The paper examines the structure of Transformers, large language models, and their potential and challenges in patent research.