Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books

A market maker provides liquidity by quoting prices to buy and sell an asset, earning the spread while managing inventory risk. We study market makers trained with multi-agent reinforcement learning on prediction markets, specifically Polymarket, whose thin order books pose distinct challenges. Three challenges underpin this research: adverse selection may cause a policy trained on real data to stop quoting; the consequences of a quote can arrive many steps after the action; and prediction-market L2 data does not identify individual orders needed for simulation. We address these challenges by reconstructing order-level LOBSTER-format data from Polymarket and training agents in JaxMARL-HFT on 417 real markets. We enrich the baseline market maker's observation space with the book and recent flow, redesign its actions and inventory controls, and propose a cooperative multi-agent decomposition into a quoter and a risk manager trained using recurrent Independent Proximal Policy Optimisation (IPPO) with RNN policies. We investigate reward shaping, particularly reward blending and post-fill markout, to mitigate credit-assignment difficulties between the agents. Our redesigned single-agent market maker quotes more frequently and incurs smaller held-out losses than the retrained JaxMARL-HFT baseline, although the contributions of jointly changed observations, actions and risk constraints are not separately identified. Decomposition alone performs poorly; careful reward design substantially improves the desk relative to the unshaped split and yields our best result. Models are evaluated on realised end-of-episode portfolio value on held-out markets. None establishes positive average portfolio value in periodic held-out evaluation, and each configuration was trained with one seed. The environment, data and protocol lay a foundation for makers that quote many linked markets at once.

1. 25 Sep 2026

   Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning

   The research extends adversarial reinforcement learning for market making to handle self-exciting order arrivals and price impact, using an LSTM module to improve robustness in complex microstructure environments.

   arXiv

   0cites
2. 7 Aug 2024

   Consistent time travel for realistic interactions with historical data: reinforcement learning for market making

   The article discusses the use of consistent data time travel in offline reinforcement learning for market making in limit order books.

   arXiv

   0cites
3. 10 Jul 2025

   Reinforcement learning for trade execution with market and limit orders

   The paper presents a reinforcement learning framework for optimal trade execution, using multivariate logistic-normal distributions, which outperforms traditional benchmark strategies.

   arXivIn Quantitative Finance

   6cites
4. 26 Feb 2025

   Event-Based Limit Order Book Simulation under a Neural Hawkes Process: Application in Market-Making

   An event-driven Limit Order Book model using a Neural Hawkes process is proposed to simulate high-frequency dynamics in financial markets, offering a more accurate depiction of trade execution.

   arXivIn Applied Mathematical Finance

   2cites
5. 31 Jul 2024

   Market Making with Exogenous Competition

   A study reveals that a 'reference market maker' who optimizes her posted depths can achieve a near perfect solution, which is compared against other solutions using an Euler scheme or reinforcement learning techniques in a competitive environment.

   arXivIn SIAM J. Financial Math.

   12cites
6. 24 Jul 2024

   The Negative Drift of a Limit Order Fill

   The research identifies a negative drift in market making models, particularly in limit order fills, using the 10 Year US Treasury Bond futures for empirical simulation.

   arXiv

   5cites