---
title: Risk-Sensitive Reinforcement Learning for an Options Market Maker: An Empirical Study with Real S&P 500 Options
url: https://www.ml-quant.com/papers/arxiv/2611.10956/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-11-14
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2611.10956
source_url: https://arxiv.org/abs/2611.10956
featured: 2026-11-13
citations: unknown
topic: Trading, Microstructure & Execution
---


# Risk-Sensitive Reinforcement Learning for an Options Market Maker: An Empirical Study with Real S&P 500 Options

A reinforcement learning agent trained on historical S&P 500 options data reduces tail losses by 3.1% versus delta hedging, reaching 8.9% in volatile downturns.

- Source: https://arxiv.org/abs/2611.10956
- Identifier: arXiv:2611.10956
- Released: 2026-11-10
- First featured: Quant Letter No. 139 (2026-11-13): https://www.ml-quant.com/issues/2026-11-13/
- Citations (Semantic Scholar): not tracked
- Published in: not yet
- Topic: Trading, Microstructure & Execution
- Authors: Jacky Chen, Ing-Haw Cheng, John Hull, Landis Li, Zissis Poulos, Justin Shi, Andreas Veneris, Zeyu Wang

## Abstract (arXiv, CC0)

Options market makers absorb client order flow and must hedge the resulting inventory risk under trading frictions. We study whether an option hedging reinforcement learning (RL) agent trained on historical market data outperforms three alternatives in tail-risk protection: (i) traditional delta hedging, (ii) simple hedge-ratio rules, and (iii) the same agent trained in simulated markets. Most RL hedging agents are trained in model-generated markets that do not capture the crashes and volatility regimes present in real data. We introduce an empirical replay environment using S&P 500 option data from 1996 to 2017 and compare against a matched geometric Brownian motion (GBM) environment, a stochastic-volatility (SABR) simulator, and a hybrid that values historical paths with Black-Scholes, all sharing the same portfolio mechanics, stochastic arrivals, costs, and test period. We train a distributional Twin Delayed Deep Deterministic Policy Gradient (TD3) agent and evaluate it out of sample over 2018-2023. The agent trained on empirical data reduces Conditional Value-at-Risk (CVaR) tail loss by 3.1% compared to delta hedging and outperforms every hedge-ratio rule chosen during the training period and every alternative training environment. The reduction reaches 8.9% in volatile and falling markets, but reverses to a cost of 7-10% in rising markets. Empirical training induces a state-dependent short net delta: the policy under-hedges negative-delta books, most strongly after prices have fallen. This tilt disappears with GBM training and is weaker with SABR or hybrid training. These findings suggest that exposure to historical market dynamics can materially change a tail-risk hedging policy, although the evidence rests on a single test path and the edge is regime-conditional.

## Related

- [Market Making with Exogenous Competition](https://www.ml-quant.com/papers/arxiv/2407.17393/): A study reveals that a 'reference market maker' who optimizes her posted depths can achieve a near perfect solution, which is compared against other solutions using an Euler scheme or reinforcement learning techniques in a competitive environment.
- [Towards Robust Reinforcement Learning Market Making: Lessons from Loss Function Experimentation](https://www.ml-quant.com/papers/arxiv/2611.09012/): Adding inventory and cash management penalties to reinforcement learning market makers improves resilience and adaptability compared to pure profit-maximizing approaches.
- [Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books](https://www.ml-quant.com/papers/arxiv/2610.26789/): Decomposing a market maker into quoter and risk-manager agents trained with cooperative IPPO on Polymarket reduces held-out losses and improves quoting frequency versus baselines.
- [Reinforcement Learning with Hawkes-Structured Networks for Market Making](https://www.ml-quant.com/papers/arxiv/2610.16524/): The study proposes a neural network architecture for market making that models self-exciting order arrivals through learnable exponential features and trains via reinforcement learning to optimize quoting policy.
- [Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning](https://www.ml-quant.com/papers/arxiv/2609.22785/): The research extends adversarial reinforcement learning for market making to handle self-exciting order arrivals and price impact, using an LSTM module to improve robustness in complex microstructure environments.
- [Consistent time travel for realistic interactions with historical data: reinforcement learning for market making](https://www.ml-quant.com/papers/arxiv/2408.02322/): The article discusses the use of consistent data time travel in offline reinforcement learning for market making in limit order books.