Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets

AI-based trading methods have rapidly evolved from machine learning and reinforcement learning to large language models (LLMs) and trading agents, yet their performance is still predominantly assessed through historical backtesting. Such evaluations provide limited evidence of whether a method can generalize to unseen future markets or whether its backtested performance can be sustained in realistic trading frictions (e.g., latency, slippage, liquidity constraints, and market impact). We present a unified benchmark that evaluates representative machine learning, reinforcement learning, LLM-based, and agent-based trading methods in cryptocurrency markets through three progressively more realistic stages: historical backtesting, prospective exchange-based paper trading, and real-money live trading. These stages jointly increase temporal realism by moving from historical to unseen future markets, and execution realism by moving from offline simulation toward live trading. This protocol enables us to quantify the backtest-to-realization gap, identify when performance begins to deteriorate, and compare how this gap differs across major classes of AI trading methods. We further provide a unified open-source system supporting all three evaluation stages, together with a public platform that continuously updates benchmark results. Code is available at https://github.com/Starlien95/Awesome-TradingAI.

1. 25 Sep 2026

   Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors

   Proposes a statistical referee that judges investment factors proposed by language-model agents using out-of-sample market outcomes, ensuring false-discovery control at any stopping time.

   arXiv

   0cites
2. 14 May 2025

   Can Generative AI agents behave like humans? Evidence from laboratory market experiments

   Large Language Models (LLMs) have potential in mimicking human behavior in economic markets, but need more research for improved diversity and accuracy.

   arXiv

   19cites
3. 18 Dec 2024

   VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback

   An auction mechanism is introduced to enhance cost-efficiency in fine-tuning large language models using Reinforcement Learning from Human Feedback, focusing on quality feedback and model performance.

   arXivIn Prima

   3cites
4. 9 Oct 2024

   MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

   The MA-RLHF framework integrates macro actions into the learning process of large language models, enhancing learning efficiency and performance in tasks like text summarization and dialogue generation.

   Machine learningIn International Conference on Learning Representations

   15cites
5. 2 Oct 2026

   AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning

   The research uses reinforcement learning with multi-objective rewards to discover formulaic alphas that work well together despite evolving reward functions and shifting environments.

   arXiv

   3fanfare
6. 2 Oct 2026

   LiveOption: Evaluating LLM Agents in Structured Option Trading with Nonlinear Payoffs

   An evaluation framework tests language model agents on structured option trading tasks, finding current systems underperform in most real-world scenarios.

   arXiv

   3fanfare