AlphaPareto: Formulaic Alpha Discovery with LLM-Guided Multi-Objective Reinforcement Learning

Formulaic alpha discovery is a core challenge in quantitative trading, as identifying alphas that work well together remains difficult. Recent reinforcement learning (RL) methods formulate this task as a Markov decision process (MDP), but two important issues remain unresolved. First, as the alpha pool evolves, the reward function changes accordingly, making the MDP inherently non-stationary. Second, most existing methods optimize a single objective, typically predictive power, while ignoring other important properties of a high-quality alpha pool. Motivated by these challenges, we propose AlphaPareto, an RL method for formulaic alpha discovery. To address non-stationarity, AlphaPareto augments the state to include both the alpha under construction and the current alpha pool, and applies a large language model (LLM) to encode the pool. This design allows the agent to adapt to the evolving search environment. To overcome the limitation of single-objective reward design, AlphaPareto replaces the scalar reward with a multi-objective vector-valued reward that simultaneously captures predictive power, temporal stability, perturbation robustness, and diversity, and optimizes these objectives through a Pareto-regularized learning procedure. Empirical applications to real-world datasets show that our AlphaPareto method outperforms its competitors.

1. 18 Dec 2024

   VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback

   An auction mechanism is introduced to enhance cost-efficiency in fine-tuning large language models using Reinforcement Learning from Human Feedback, focusing on quality feedback and model performance.

   arXivIn Prima

   3cites
2. 9 Oct 2024

   MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions

   The MA-RLHF framework integrates macro actions into the learning process of large language models, enhancing learning efficiency and performance in tasks like text summarization and dialogue generation.

   Machine learningIn International Conference on Learning Representations

   15cites
3. 2 Oct 2026

   Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets

   A benchmark compares machine learning, reinforcement learning, and large language model trading methods across historical backtests, paper trading, and live markets to measure the gap.

   arXiv

   4fanfare
4. 24 Apr 2024

   Mastering Diverse Domains through World Models

   Algorithm Mastery: DreamerV3, a universal algorithm, excels in over 150 varied tasks, including diamond collection in Minecraft without human input, expanding the scope of reinforcement learning.

   Machine learning

   1,451cites
5. 10 Jul 2024

   SimPO: Simple Preference Optimization with a Reference-Free Reward

   Simple Preference Optimization: SimPO improves reinforcement learning from human feedback by using the average log probability of a sequence as the implicit reward, enhancing training stability and computational efficiency.

   Machine learningIn Neural Information Processing Systems

   1,198cites
6. 15 May 2024

   AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments

   AI Evaluation in Clinical Environments: The paper introduces AgentClinic, a benchmark for assessing large language models in simulated clinical environments, highlighting the significant impact of biases on diagnostic accuracy and patient interactions.

   Machine learning

   263cites